[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:40 dist-test-slave-fbq8 krb5kdc[822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:40 dist-test-slave-fbq8 krb5kdc[822](info): set up 2 sockets
Jun 19 11:59:40 dist-test-slave-fbq8 krb5kdc[822](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 11:59:41.130179 815 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:41 dist-test-slave-fbq8 krb5kdc[822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:41.277455 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42711
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:41.365198 835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:41.365499 835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:41.365540 835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:41.366719 835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:41.366811 835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:41.366837 835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:41.366856 835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:41.366882 835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:41.368438 835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:41.368786 835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:41.369064 835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:41.375119 841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:41.375128 843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:41.375109 840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:41.375375 835 server_base.cc:1034] running on GCE node
I20260619 11:59:41.375716 835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:41.376019 835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:41.377218 835 hybrid_clock.cc:648] HybridClock initialized: now 1781870381377195 us; error 43 us; skew 500 ppm
Jun 19 11:59:41 dist-test-slave-fbq8 krb5kdc[822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:41.380235 835 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:41.380957 835 webserver.cc:460] Webserver started at http://127.0.203.254:38289/ using document root <none> and password file <none>
I20260619 11:59:41.381266 835 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:41.381390 835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:41.381582 835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:41.382866 835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "f4424bd3bdea4649a5793ec37000cb7b"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "68b55df058deac457815e84e70efcb6a"
server_key_iv: "2f3e2c23ed95191bd0d36b7a7a8a16b5"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.383344 835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "f4424bd3bdea4649a5793ec37000cb7b"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "68b55df058deac457815e84e70efcb6a"
server_key_iv: "2f3e2c23ed95191bd0d36b7a7a8a16b5"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.385278 835 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:59:41.386255 850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:41.386629 835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:41.386770 835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "f4424bd3bdea4649a5793ec37000cb7b"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "68b55df058deac457815e84e70efcb6a"
server_key_iv: "2f3e2c23ed95191bd0d36b7a7a8a16b5"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.386842 835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:41.414402 835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:41.416983 835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:41.417186 835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:41.422638 835 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:42711
I20260619 11:59:41.422794 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:42711 every 8 connection(s)
I20260619 11:59:41.423377 835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:41.424057 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 835
I20260619 11:59:41.424196 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:41.424324 903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:41.425028 815 external_mini_cluster.cc:1428] Setting key 429f77da72f4866f523fc2645ac5e140
I20260619 11:59:41.427284 903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b: Bootstrap starting.
I20260619 11:59:41.427965 903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:41.428309 903 log.cc:824] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:41.429153 903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b: No bootstrap required, opened a new log
I20260619 11:59:41.432314 903 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4424bd3bdea4649a5793ec37000cb7b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42711 } }
I20260619 11:59:41.432514 903 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:41.432559 903 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4424bd3bdea4649a5793ec37000cb7b, State: Initialized, Role: FOLLOWER
I20260619 11:59:41.432789 903 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4424bd3bdea4649a5793ec37000cb7b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42711 } }
I20260619 11:59:41.432902 903 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:41.432951 903 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:41.433005 903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:41.433773 903 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4424bd3bdea4649a5793ec37000cb7b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42711 } }
I20260619 11:59:41.433911 903 leader_election.cc:302] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4424bd3bdea4649a5793ec37000cb7b; no voters:
I20260619 11:59:41.434108 903 leader_election.cc:288] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:41.434194 908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:41.434422 903 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:41.434448 908 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [term 1 LEADER]: Becoming Leader. State: Replica: f4424bd3bdea4649a5793ec37000cb7b, State: Running, Role: LEADER
I20260619 11:59:41.434742 908 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4424bd3bdea4649a5793ec37000cb7b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42711 } }
I20260619 11:59:41.435527 909 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4424bd3bdea4649a5793ec37000cb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4424bd3bdea4649a5793ec37000cb7b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42711 } } }
I20260619 11:59:41.435634 909 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:41.435890 910 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4424bd3bdea4649a5793ec37000cb7b. Latest consensus state: current_term: 1 leader_uuid: "f4424bd3bdea4649a5793ec37000cb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4424bd3bdea4649a5793ec37000cb7b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42711 } } }
I20260619 11:59:41.435959 910 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b [sys.catalog]: This master's current role is: LEADER
W20260619 11:59:41.436785 923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:59:41.436856 923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:59:41.436968 918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:41.437234 918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 11:59:41 dist-test-slave-fbq8 krb5kdc[822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:41.442385 918 catalog_manager.cc:1347] Generated new cluster ID: a243bb6e2e1445a093a5f9f8b57eb727
I20260619 11:59:41.442449 918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:59:41.446849 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:41.426136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.0.203.254:42711)
0619 11:59:41.426520 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:41.426529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:41.426570 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:59:41.429172 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:41.429179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:41.429235 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:41.429422 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:41.432161 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.433409 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.435443 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.436191 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.436323 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:41.441453 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:41.441459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:41.441460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:41.441475 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:41.443721 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:41.444432 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:41.444436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:41.444437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:41.444561 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:41.445481 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:41.445482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:41.445482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:41.445838 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:41.446026 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:41.446413 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 11:59:41.446573 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":113,"threads_started":1}
I20260619 11:59:41.464165 918 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:41.465116 918 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:41.469988 918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4424bd3bdea4649a5793ec37000cb7b: Generated new TSK 0
I20260619 11:59:41.470235 918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 11:59:41.514328 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42711
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:41.619063 929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:41.619298 929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:41.619326 929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:41.620795 929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:41.620864 929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:41.620896 929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 11:59:41.622470 929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:41.622694 929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:41.622895 929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:41.628755 935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:41.628827 934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:41.628755 937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:41.628890 929 server_base.cc:1034] running on GCE node
I20260619 11:59:41.629449 929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:41.629729 929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:41.630931 929 hybrid_clock.cc:648] HybridClock initialized: now 1781870381630899 us; error 50 us; skew 500 ppm
Jun 19 11:59:41 dist-test-slave-fbq8 krb5kdc[822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:41.633986 929 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 11:59:41.634564 929 webserver.cc:460] Webserver started at http://127.0.203.193:35017/ using document root <none> and password file <none>
I20260619 11:59:41.634862 929 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:41.634944 929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:41.635090 929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:41.636308 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "e74ca13d6c6e454f9573b80af99d74ef"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "42c2fb55f44f2e024ba3b96192b1393a"
server_key_iv: "fc743baf50221c1b919f5a268107fdc0"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.637159 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "e74ca13d6c6e454f9573b80af99d74ef"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "42c2fb55f44f2e024ba3b96192b1393a"
server_key_iv: "fc743baf50221c1b919f5a268107fdc0"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.638629 929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:41.639647 944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:41.639978 929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:41.640100 929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "e74ca13d6c6e454f9573b80af99d74ef"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "42c2fb55f44f2e024ba3b96192b1393a"
server_key_iv: "fc743baf50221c1b919f5a268107fdc0"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.640182 929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:41.653157 929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:41.655730 929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:41.655941 929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:41.656200 929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:41.656569 929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:41.656636 929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:41.656706 929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:41.656733 929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:41.665633 929 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:33979
I20260619 11:59:41.665678 1057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:33979 every 8 connection(s)
I20260619 11:59:41.666059 929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 11:59:41.672731 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 929
I20260619 11:59:41.672856 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 11:59:41.673043 815 external_mini_cluster.cc:1428] Setting key 68e8d17fde6504286189934bb89b1310
Jun 19 11:59:41 dist-test-slave-fbq8 krb5kdc[822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:41.680322 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:41.667073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:41687 (local address 127.0.203.254:42711)
0619 11:59:41.667215 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:41.667218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:41.667429 (+ 211us) server_negotiation.cc:407] Connection header received
0619 11:59:41.668098 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:41.668099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:41.668115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:41.668175 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:41.669470 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.670142 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.671308 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.671549 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.671663 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:41.675559 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:41.675561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:41.675562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:41.675571 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:41.678087 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:41.678833 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:41.678834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:41.678835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:41.678915 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:41.679276 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:41.679277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:41.679277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:41.679477 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:41.679585 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:41.680026 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 11:59:41.680157 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 11:59:41.680356 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:41.666981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:42711 (local address 127.0.203.193:41687)
0619 11:59:41.667251 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:41.667309 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:41.667959 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:41.668273 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:41.668278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:41.669063 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:41.669317 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:41.669323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.670294 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.670296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:41.671198 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:41.671202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.671674 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.671675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:41.671826 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:41.672552 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:41.672564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:41.675360 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:41.678250 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:41.678262 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:41.678265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:41.678690 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:41.679044 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:41.679045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:41.679045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:41.679175 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:41.679628 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:41.679634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:41.679846 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 11:59:41.679970 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 11:59:41.680149 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
I20260619 11:59:41.681528 1058 heartbeater.cc:347] Connected to a master server at 127.0.203.254:42711
I20260619 11:59:41.681685 1058 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:41.681915 1058 heartbeater.cc:511] Master 127.0.203.254:42711 requested a full tablet report, sending...
I20260619 11:59:41.682885 867 ts_manager.cc:194] Registered new tserver with Master: e74ca13d6c6e454f9573b80af99d74ef (127.0.203.193:33979)
I20260619 11:59:41.683588 867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:41687
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 11:59:41.704804 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42711
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:41.797019 1063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:41.797312 1063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:41.797350 1063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:41.798666 1063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:41.798746 1063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:41.798794 1063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 11:59:41.800361 1063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:41.800694 1063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:41.800976 1063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:41.806821 1071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:41.806841 1069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:41.807126 1068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:41.807643 1063 server_base.cc:1034] running on GCE node
I20260619 11:59:41.807919 1063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:41.808219 1063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:41.809407 1063 hybrid_clock.cc:648] HybridClock initialized: now 1781870381809377 us; error 48 us; skew 500 ppm
Jun 19 11:59:41 dist-test-slave-fbq8 krb5kdc[822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:41.812805 1063 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 11:59:41.813444 1063 webserver.cc:460] Webserver started at http://127.0.203.194:46423/ using document root <none> and password file <none>
I20260619 11:59:41.813722 1063 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:41.813798 1063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:41.813964 1063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:41.815183 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "c77e233f955f47a59e1cd4aa27299a8d"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "3d8ca49c857c60dac1f13a3debca7a61"
server_key_iv: "03fdd0452425507efdd7f70a67468c12"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.815600 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "c77e233f955f47a59e1cd4aa27299a8d"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "3d8ca49c857c60dac1f13a3debca7a61"
server_key_iv: "03fdd0452425507efdd7f70a67468c12"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.817318 1063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 11:59:41.818368 1078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:41.818686 1063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:41.818809 1063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "c77e233f955f47a59e1cd4aa27299a8d"
format_stamp: "Formatted at 2026-06-19 11:59:41 on dist-test-slave-fbq8"
server_key: "3d8ca49c857c60dac1f13a3debca7a61"
server_key_iv: "03fdd0452425507efdd7f70a67468c12"
server_key_version: "encryptionkey@0"
I20260619 11:59:41.818907 1063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:41.856190 1063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:41.859190 1063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:41.859396 1063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:41.859737 1063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:41.860160 1063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:41.860232 1063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:41.860270 1063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:41.860299 1063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:41.869843 1063 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:43927
I20260619 11:59:41.869899 1191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:43927 every 8 connection(s)
I20260619 11:59:41.870353 1063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 11:59:41.871507 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 1063
I20260619 11:59:41.871659 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 11:59:41.871853 815 external_mini_cluster.cc:1428] Setting key 17a68eb6af564af0ebdb1017c1e0504b
Jun 19 11:59:41 dist-test-slave-fbq8 krb5kdc[822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
I20260619 11:59:41.884320 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:41.872450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:40273 (local address 127.0.203.254:42711)
0619 11:59:41.872544 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:41.872547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:41.872560 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:59:41.873169 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:41.873170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:41.873182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:41.873243 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:41.874332 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.875086 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.876419 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.876813 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.876912 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:41.879802 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:41.879804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:41.879805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:41.879816 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:41.882053 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:41.882658 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:41.882660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:41.882660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:41.882729 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:41.883200 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:41.883201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:41.883202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:41.883336 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:41.883403 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:41.884047 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 11:59:41.884159 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:59:41.884550 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:41.872076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:42711 (local address 127.0.203.194:40273)
0619 11:59:41.872270 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:41.872321 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:41.873030 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:41.873328 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:41.873332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:41.873982 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:41.874215 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:41.874220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.875234 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.875235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:41.876270 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:41.876273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:41.876944 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:41.876945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:41.877071 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:41.877741 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:41.877751 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:41.879639 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:41.882202 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:41.882214 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:41.882217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:41.882543 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:41.882912 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:41.882914 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:41.882914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:41.883097 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:41.883487 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:41.883494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:41.883759 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 11:59:41.883886 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 11:59:41.884216 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260619 11:59:41.885785 1192 heartbeater.cc:347] Connected to a master server at 127.0.203.254:42711
I20260619 11:59:41.885928 1192 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:41.886194 1192 heartbeater.cc:511] Master 127.0.203.254:42711 requested a full tablet report, sending...
I20260619 11:59:41.886765 867 ts_manager.cc:194] Registered new tserver with Master: c77e233f955f47a59e1cd4aa27299a8d (127.0.203.194:43927)
I20260619 11:59:41.887373 867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:40273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 11:59:41.904628 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42711
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:42.002035 1197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:42.002334 1197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:42.002379 1197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:42.003674 1197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:42.003767 1197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:42.003819 1197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 11:59:42.005573 1197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:42.005885 1197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:42.006150 1197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:42.011981 1202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:42.011981 1203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:42.012146 1197 server_base.cc:1034] running on GCE node
W20260619 11:59:42.012058 1205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:42.012516 1197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:42.012866 1197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:42.014067 1197 hybrid_clock.cc:648] HybridClock initialized: now 1781870382014042 us; error 54 us; skew 500 ppm
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:42.017079 1197 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 11:59:42.017621 1197 webserver.cc:460] Webserver started at http://127.0.203.195:43551/ using document root <none> and password file <none>
I20260619 11:59:42.017876 1197 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:42.017947 1197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:42.018105 1197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:42.019199 1197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "28ffc47d46c849ad9d1be6c7f4365c64"
format_stamp: "Formatted at 2026-06-19 11:59:42 on dist-test-slave-fbq8"
server_key: "b8d6728f91961a9341a3346e8c5e8a90"
server_key_iv: "5745a47ba182b6bd8b35b6ab6cb5a51b"
server_key_version: "encryptionkey@0"
I20260619 11:59:42.019578 1197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "28ffc47d46c849ad9d1be6c7f4365c64"
format_stamp: "Formatted at 2026-06-19 11:59:42 on dist-test-slave-fbq8"
server_key: "b8d6728f91961a9341a3346e8c5e8a90"
server_key_iv: "5745a47ba182b6bd8b35b6ab6cb5a51b"
server_key_version: "encryptionkey@0"
I20260619 11:59:42.021196 1197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 11:59:42.022118 1212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:42.022416 1197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:42.022535 1197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "28ffc47d46c849ad9d1be6c7f4365c64"
format_stamp: "Formatted at 2026-06-19 11:59:42 on dist-test-slave-fbq8"
server_key: "b8d6728f91961a9341a3346e8c5e8a90"
server_key_iv: "5745a47ba182b6bd8b35b6ab6cb5a51b"
server_key_version: "encryptionkey@0"
I20260619 11:59:42.022647 1197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:42.036813 1197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:42.039178 1197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:42.039310 1197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:42.039490 1197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:42.039829 1197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:42.039875 1197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:42.039899 1197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:42.039953 1197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:42.047423 1197 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:34885
I20260619 11:59:42.047520 1325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:34885 every 8 connection(s)
I20260619 11:59:42.047868 1197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 11:59:42.051085 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 1197
I20260619 11:59:42.051162 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 11:59:42.051365 815 external_mini_cluster.cc:1428] Setting key 92fc58a5bbbc30b96b891e44a674a0ba
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:42.059127 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:42.048780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:48079 (local address 127.0.203.254:42711)
0619 11:59:42.048836 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:42.048839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:42.049093 (+ 254us) server_negotiation.cc:407] Connection header received
0619 11:59:42.049751 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:42.049752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:42.049766 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:42.049816 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:42.050935 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.051643 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.053310 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.053542 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.053626 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:42.056201 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:42.056203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:42.056203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:42.056209 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:42.057783 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:42.058199 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:42.058200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:42.058200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:42.058255 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:42.058499 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:42.058500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:42.058500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:42.058617 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:42.058674 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:42.058958 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 11:59:42.059034 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 11:59:42.059286 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:42.048738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:42711 (local address 127.0.203.195:48079)
0619 11:59:42.048910 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:42.048966 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:42.049620 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:42.049897 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:42.049908 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:42.050571 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:42.050798 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:42.050803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.051811 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.051814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:42.053192 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:42.053195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.053652 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.053653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:42.053786 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:42.054392 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:42.054401 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:42.056095 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:42.057868 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:42.057874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:42.057876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:42.058113 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:42.058327 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:42.058327 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:42.058328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:42.058432 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:42.058684 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:42.058688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:42.058807 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 11:59:42.058940 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 11:59:42.059072 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260619 11:59:42.059834 1326 heartbeater.cc:347] Connected to a master server at 127.0.203.254:42711
I20260619 11:59:42.059976 1326 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:42.060225 1326 heartbeater.cc:511] Master 127.0.203.254:42711 requested a full tablet report, sending...
I20260619 11:59:42.060765 867 ts_manager.cc:194] Registered new tserver with Master: 28ffc47d46c849ad9d1be6c7f4365c64 (127.0.203.195:34885)
I20260619 11:59:42.061213 867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:48079
I20260619 11:59:42.064155 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 11:59:42.092391 1333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:42.082828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.0.203.193:33979)
0619 11:59:42.083048 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:42.083054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:42.083065 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:59:42.083095 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:42.083102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:42.083119 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:42.083220 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:42.083859 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.084726 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.085519 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.085872 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.085973 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:42.088933 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:42.088942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:42.088943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:42.088950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:42.090926 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:42.091474 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:42.091478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:42.091478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:42.091557 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:42.091801 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:42.091801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:42.091802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:42.091950 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:42.092078 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:42.092177 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 11:59:42.092242 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
W20260619 11:59:42.092952 972 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49712
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 11:59:42.121948 1333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:42.111763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.0.203.193:33979)
0619 11:59:42.111910 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:42.111913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:42.112036 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:59:42.112144 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:42.112145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:42.112159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:42.112234 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:42.113155 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.113687 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.114467 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:42.114681 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:42.114801 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:42.117840 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:42.117843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:42.117843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:42.117857 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:42.119966 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:42.120571 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:42.120572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:42.120573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:42.120700 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:42.121228 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:42.121229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:42.121229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:42.121372 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:42.121496 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:42.121731 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 11:59:42.121865 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 11:59:42.123095 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 929
I20260619 11:59:42.129163 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 1063
I20260619 11:59:42.135588 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 1197
I20260619 11:59:42.141003 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 835
2026-06-19T11:59:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2115 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[1340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[1340](info): set up 2 sockets
Jun 19 11:59:42 dist-test-slave-fbq8 krb5kdc[1340](info): commencing operation
krb5kdc: starting...
W20260619 11:59:43.228333 815 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:43 dist-test-slave-fbq8 krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:43.367945 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33231
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:33231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:43.459784 1353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:43.460048 1353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:43.460073 1353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:43.461371 1353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:43.461419 1353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:43.461434 1353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:43.461447 1353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:43.461459 1353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:43.463013 1353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:33231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:43.463331 1353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:43.463587 1353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:43.469256 1359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:43.469246 1361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:43.469246 1358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:43.469514 1353 server_base.cc:1034] running on GCE node
I20260619 11:59:43.469859 1353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:43.470160 1353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:43.471414 1353 hybrid_clock.cc:648] HybridClock initialized: now 1781870383471379 us; error 57 us; skew 500 ppm
Jun 19 11:59:43 dist-test-slave-fbq8 krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:43.474817 1353 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:43.475499 1353 webserver.cc:460] Webserver started at http://127.0.203.254:33039/ using document root <none> and password file <none>
I20260619 11:59:43.476032 1353 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:43.476174 1353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:43.476370 1353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:43.477824 1353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "56bb1ce63a424bbe817cfa746808e39a"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "cdcbfccccc2f92b595c77727da9e9bfc"
server_key_iv: "9ba06db74fb1fb326375e3dd28cc2b91"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.478258 1353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "56bb1ce63a424bbe817cfa746808e39a"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "cdcbfccccc2f92b595c77727da9e9bfc"
server_key_iv: "9ba06db74fb1fb326375e3dd28cc2b91"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.479859 1353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 11:59:43.481057 1368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:43.481432 1353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:43.481603 1353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "56bb1ce63a424bbe817cfa746808e39a"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "cdcbfccccc2f92b595c77727da9e9bfc"
server_key_iv: "9ba06db74fb1fb326375e3dd28cc2b91"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.481789 1353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:43.503037 1353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:43.505849 1353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:43.506050 1353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:43.511409 1353 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:33231
I20260619 11:59:43.511448 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:33231 every 8 connection(s)
I20260619 11:59:43.511857 1353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:43.513212 1421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:43.514338 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 1353
I20260619 11:59:43.514433 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:43.514734 815 external_mini_cluster.cc:1428] Setting key e7e1d6e6e605b89fbfed5d0df0b4b1d6
I20260619 11:59:43.516340 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a: Bootstrap starting.
I20260619 11:59:43.517289 1421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:43.517688 1421 log.cc:824] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:43.518743 1421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a: No bootstrap required, opened a new log
I20260619 11:59:43.521270 1421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb1ce63a424bbe817cfa746808e39a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33231 } }
I20260619 11:59:43.521500 1421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:43.521560 1421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bb1ce63a424bbe817cfa746808e39a, State: Initialized, Role: FOLLOWER
I20260619 11:59:43.521775 1421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb1ce63a424bbe817cfa746808e39a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33231 } }
I20260619 11:59:43.521899 1421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:43.521958 1421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:43.522019 1421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:43.522948 1421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb1ce63a424bbe817cfa746808e39a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33231 } }
I20260619 11:59:43.523087 1421 leader_election.cc:302] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56bb1ce63a424bbe817cfa746808e39a; no voters:
Jun 19 11:59:43 dist-test-slave-fbq8 krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:43.523285 1421 leader_election.cc:288] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:43.523355 1426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:43.523499 1426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [term 1 LEADER]: Becoming Leader. State: Replica: 56bb1ce63a424bbe817cfa746808e39a, State: Running, Role: LEADER
I20260619 11:59:43.523583 1426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb1ce63a424bbe817cfa746808e39a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33231 } }
I20260619 11:59:43.523614 1421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:43.523954 1426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56bb1ce63a424bbe817cfa746808e39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb1ce63a424bbe817cfa746808e39a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33231 } } }
I20260619 11:59:43.524067 1426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:43.524238 1427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56bb1ce63a424bbe817cfa746808e39a. Latest consensus state: current_term: 1 leader_uuid: "56bb1ce63a424bbe817cfa746808e39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb1ce63a424bbe817cfa746808e39a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33231 } } }
I20260619 11:59:43.524533 1427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:43.524781 1432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:43.525583 1432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:59:43.528573 1432 catalog_manager.cc:1347] Generated new cluster ID: 3d79dcde6cc548258d1e4b7f84b26bf6
I20260619 11:59:43.528635 1432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:59:43.528903 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:43.515487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.0.203.254:33231)
0619 11:59:43.515728 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:43.515738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:43.515770 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:59:43.516692 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:43.516704 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:43.516784 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:43.516968 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:43.517740 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.519063 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.519873 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.520353 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.520491 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:43.523946 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:43.523953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:43.523953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:43.523976 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:43.526528 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:43.527151 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:43.527155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:43.527155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:43.527284 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:43.527592 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:43.527595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:43.527595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:43.528001 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:43.528198 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:43.528387 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 11:59:43.528599 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":117,"threads_started":1}
I20260619 11:59:43.545254 1432 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:43.545943 1432 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:43.553231 1432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56bb1ce63a424bbe817cfa746808e39a: Generated new TSK 0
I20260619 11:59:43.553490 1432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 11:59:43.586143 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33231
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:43.680423 1447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:43.680738 1447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:43.680768 1447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:43.682406 1447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:43.682475 1447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:43.682505 1447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 11:59:43.684052 1447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:43.684301 1447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:43.684526 1447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:43.690961 1452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:43.691171 1447 server_base.cc:1034] running on GCE node
W20260619 11:59:43.690961 1453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:43.691048 1455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:43.691489 1447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:43.691821 1447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:43.693053 1447 hybrid_clock.cc:648] HybridClock initialized: now 1781870383693021 us; error 50 us; skew 500 ppm
Jun 19 11:59:43 dist-test-slave-fbq8 krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:43.696241 1447 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 11:59:43.696918 1447 webserver.cc:460] Webserver started at http://127.0.203.193:45099/ using document root <none> and password file <none>
I20260619 11:59:43.697192 1447 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:43.697275 1447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:43.697445 1447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:43.698608 1447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "749a1436d010431fa433ab4d5fda6042"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "7def15ef93b0244e215b60ab9ba489f4"
server_key_iv: "f222fd238384f28081ac01a5a600e358"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.699020 1447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "749a1436d010431fa433ab4d5fda6042"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "7def15ef93b0244e215b60ab9ba489f4"
server_key_iv: "f222fd238384f28081ac01a5a600e358"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.700758 1447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 11:59:43.701763 1462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:43.702157 1447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:43.702301 1447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "749a1436d010431fa433ab4d5fda6042"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "7def15ef93b0244e215b60ab9ba489f4"
server_key_iv: "f222fd238384f28081ac01a5a600e358"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.702423 1447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:43.712320 1447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:43.715323 1447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:43.715543 1447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:43.715827 1447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:43.716192 1447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:43.716230 1447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:43.716291 1447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:43.716315 1447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:43.724141 1447 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:39383
I20260619 11:59:43.724267 1575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:39383 every 8 connection(s)
I20260619 11:59:43.724735 1447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:59:43 dist-test-slave-fbq8 krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:43.732825 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 1447
I20260619 11:59:43.732921 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 11:59:43.733137 815 external_mini_cluster.cc:1428] Setting key 57c53fc5b99a0e640b714a81b18ea3de
I20260619 11:59:43.737169 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:43.725952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:41195 (local address 127.0.203.254:33231)
0619 11:59:43.726084 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:43.726087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:43.726300 (+ 213us) server_negotiation.cc:407] Connection header received
0619 11:59:43.727054 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:43.727056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:43.727069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:43.727122 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:43.728204 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.728855 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.730096 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.730340 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.730439 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:43.733443 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:43.733446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:43.733446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:43.733460 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:43.735586 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:43.736062 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:43.736063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:43.736064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:43.736131 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:43.736413 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:43.736414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:43.736414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:43.736562 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:43.736639 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:43.736974 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 11:59:43.737068 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 11:59:43.737413 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:43.725811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33231 (local address 127.0.203.193:41195)
0619 11:59:43.726127 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:43.726186 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:43.726890 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:43.727203 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:43.727209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:43.727860 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:43.728080 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:43.728085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.728980 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.728981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:43.729869 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:43.729872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.730453 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.730454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:43.730606 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:43.731300 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:43.731316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:43.733259 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:43.735688 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:43.735698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:43.735702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:43.735964 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:43.736210 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:43.736210 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:43.736211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:43.736334 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:43.736668 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:43.736675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:43.736829 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 11:59:43.736954 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 11:59:43.737135 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":156,"threads_started":1}
I20260619 11:59:43.737948 1576 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33231
I20260619 11:59:43.738112 1576 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:43.738386 1576 heartbeater.cc:511] Master 127.0.203.254:33231 requested a full tablet report, sending...
I20260619 11:59:43.739060 1385 ts_manager.cc:194] Registered new tserver with Master: 749a1436d010431fa433ab4d5fda6042 (127.0.203.193:39383)
I20260619 11:59:43.739735 1385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:41195
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 11:59:43.764778 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33231
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:43.870708 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:43.871093 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:43.871131 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:43.872604 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:43.872809 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:43.872852 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 11:59:43.874758 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:43.875021 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:43.875265 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:43.881433 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:43.881428 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:43.881560 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:43.881742 1581 server_base.cc:1034] running on GCE node
I20260619 11:59:43.881999 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:43.882264 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:43.883426 1581 hybrid_clock.cc:648] HybridClock initialized: now 1781870383883413 us; error 44 us; skew 500 ppm
Jun 19 11:59:43 dist-test-slave-fbq8 krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:43.886871 1581 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 11:59:43.887452 1581 webserver.cc:460] Webserver started at http://127.0.203.194:45369/ using document root <none> and password file <none>
I20260619 11:59:43.887723 1581 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:43.887775 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:43.887946 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:43.889207 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "f5e7a824297d42719ce5516cc16c0bb2"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "2c180880d3ea1c9d16125959159fbd0e"
server_key_iv: "b39d8a5203779ac108368ef6b3617ead"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.889604 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "f5e7a824297d42719ce5516cc16c0bb2"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "2c180880d3ea1c9d16125959159fbd0e"
server_key_iv: "b39d8a5203779ac108368ef6b3617ead"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.891177 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 11:59:43.892103 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:43.892438 1581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:43.892560 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "f5e7a824297d42719ce5516cc16c0bb2"
format_stamp: "Formatted at 2026-06-19 11:59:43 on dist-test-slave-fbq8"
server_key: "2c180880d3ea1c9d16125959159fbd0e"
server_key_iv: "b39d8a5203779ac108368ef6b3617ead"
server_key_version: "encryptionkey@0"
I20260619 11:59:43.892690 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:43.914523 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:43.917186 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:43.917392 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:43.917685 1581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:43.918085 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:43.918150 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:43.918187 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:43.918215 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:43.926237 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:44999
I20260619 11:59:43.926332 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:44999 every 8 connection(s)
I20260619 11:59:43.926712 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 11:59:43.932476 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 1581
I20260619 11:59:43.932694 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 11:59:43.932920 815 external_mini_cluster.cc:1428] Setting key 063222aaf9c036b73c3873733fb59724
Jun 19 11:59:43 dist-test-slave-fbq8 krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:43.939023 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:43.927771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:53969 (local address 127.0.203.254:33231)
0619 11:59:43.928071 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:43.928074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:43.928310 (+ 236us) server_negotiation.cc:407] Connection header received
0619 11:59:43.929003 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:43.929005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:43.929020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:43.929079 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:43.930244 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.930799 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.931922 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.932130 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.932225 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:43.935242 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:43.935244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:43.935244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:43.935253 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:43.937064 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:43.937717 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:43.937719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:43.937719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:43.937801 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:43.938198 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:43.938199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:43.938199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:43.938358 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:43.938454 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:43.938802 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 11:59:43.938912 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260619 11:59:43.939399 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:43.927651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33231 (local address 127.0.203.194:53969)
0619 11:59:43.928104 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:43.928175 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:43.928868 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:43.929178 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:43.929183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:43.929871 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:43.930121 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:43.930125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.930911 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.930912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:43.931816 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:43.931818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:43.932251 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:43.932252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:43.932395 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:43.933219 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:43.933233 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:43.935110 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:43.937196 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:43.937207 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:43.937210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:43.937603 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:43.937927 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:43.937930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:43.937930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:43.938105 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:43.938441 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:43.938449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:43.938626 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 11:59:43.938770 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 11:59:43.938988 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":124,"threads_started":1}
I20260619 11:59:43.940192 1710 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33231
I20260619 11:59:43.940312 1710 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:43.940554 1710 heartbeater.cc:511] Master 127.0.203.254:33231 requested a full tablet report, sending...
I20260619 11:59:43.941221 1385 ts_manager.cc:194] Registered new tserver with Master: f5e7a824297d42719ce5516cc16c0bb2 (127.0.203.194:44999)
I20260619 11:59:43.941751 1385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:53969
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 11:59:43.970192 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33231
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:44.078656 1715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:44.078959 1715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:44.078995 1715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:44.080603 1715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:44.080753 1715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:44.080833 1715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 11:59:44.082702 1715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:44.083073 1715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:44.083403 1715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:44.089675 1723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:44.089663 1720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:44.089859 1715 server_base.cc:1034] running on GCE node
W20260619 11:59:44.089658 1721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:44.090353 1715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:44.090718 1715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:44.091878 1715 hybrid_clock.cc:648] HybridClock initialized: now 1781870384091851 us; error 54 us; skew 500 ppm
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:44.095043 1715 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 11:59:44.095773 1715 webserver.cc:460] Webserver started at http://127.0.203.195:35665/ using document root <none> and password file <none>
I20260619 11:59:44.096073 1715 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:44.096129 1715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:44.096287 1715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:44.097774 1715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4"
format_stamp: "Formatted at 2026-06-19 11:59:44 on dist-test-slave-fbq8"
server_key: "20dfda90c62874882572c3d5599d6968"
server_key_iv: "a18d50d4e0f5d94375a6cf77b6d4bbe5"
server_key_version: "encryptionkey@0"
I20260619 11:59:44.098201 1715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4"
format_stamp: "Formatted at 2026-06-19 11:59:44 on dist-test-slave-fbq8"
server_key: "20dfda90c62874882572c3d5599d6968"
server_key_iv: "a18d50d4e0f5d94375a6cf77b6d4bbe5"
server_key_version: "encryptionkey@0"
I20260619 11:59:44.100306 1715 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260619 11:59:44.101893 1730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:44.102298 1715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:44.102434 1715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4"
format_stamp: "Formatted at 2026-06-19 11:59:44 on dist-test-slave-fbq8"
server_key: "20dfda90c62874882572c3d5599d6968"
server_key_iv: "a18d50d4e0f5d94375a6cf77b6d4bbe5"
server_key_version: "encryptionkey@0"
I20260619 11:59:44.102550 1715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:44.126552 1715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:44.129568 1715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:44.129734 1715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:44.130133 1715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:44.130786 1715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:44.131021 1715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:44.131141 1715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:44.131179 1715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:44.139078 1715 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:44757
I20260619 11:59:44.139170 1843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:44757 every 8 connection(s)
I20260619 11:59:44.139571 1715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:44.147912 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 1715
I20260619 11:59:44.148051 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 11:59:44.148262 815 external_mini_cluster.cc:1428] Setting key 0af5f0baec025ea20f58e9ff73b74342
I20260619 11:59:44.152580 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.140599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:43149 (local address 127.0.203.254:33231)
0619 11:59:44.140731 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.140734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.141088 (+ 354us) server_negotiation.cc:407] Connection header received
0619 11:59:44.141765 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.141766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.141780 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.141863 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:44.143105 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.143739 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.144976 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.145306 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.145418 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.148424 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:44.148427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:44.148427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:44.148438 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:44.150618 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.151332 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.151333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.151333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.151403 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.151807 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.151809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.151809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.151976 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:44.152046 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.152424 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.152510 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 11:59:44.153007 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.140512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33231 (local address 127.0.203.195:43149)
0619 11:59:44.140851 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:44.140918 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:44.141597 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:44.141964 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:44.141968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:44.142728 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:44.142970 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.142976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.143920 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.143921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.144855 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.144858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.145519 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.145520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.145652 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.146305 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:44.146315 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:44.148237 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:44.150821 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:44.150831 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:44.150834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:44.151191 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:44.151521 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:44.151522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:44.151523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:44.151698 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:44.152082 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:44.152088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:44.152269 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 11:59:44.152390 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 11:59:44.152598 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":185,"threads_started":1}
I20260619 11:59:44.153733 1844 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33231
I20260619 11:59:44.154050 1844 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:44.154407 1844 heartbeater.cc:511] Master 127.0.203.254:33231 requested a full tablet report, sending...
I20260619 11:59:44.155089 1385 ts_manager.cc:194] Registered new tserver with Master: 4fa70d62ee584c499c8aaeb41a2b5aa4 (127.0.203.195:44757)
I20260619 11:59:44.155498 1385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:43149
I20260619 11:59:44.161584 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:59:44.171286 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.163778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.0.203.254:33231)
0619 11:59:44.163867 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.163870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.164205 (+ 335us) server_negotiation.cc:407] Connection header received
0619 11:59:44.164239 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.164240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.164253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.164327 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:44.165394 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.165923 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.166822 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.167029 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.167117 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.168535 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:44.168536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:44.168536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:44.168545 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:44.170059 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.170448 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.170449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.170449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.170509 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.170703 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.170703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.170704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.170860 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:44.170950 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.171081 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.171153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:59:44.174484 1385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:59:44.176277 1385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:44.185515 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.180887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.0.203.194:44999)
0619 11:59:44.181198 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.181204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.181216 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:59:44.181260 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.181262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.181282 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.181417 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:44.181761 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.182711 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.184252 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.185014 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.185108 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.185174 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.185371 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.185415 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":159,"threads_started":1}
I20260619 11:59:44.185505 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.180611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:44999 (local address 127.0.0.1:51040)
0619 11:59:44.180847 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:44.180898 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:44.180978 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:44.181486 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:44.181490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:44.181493 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:44.181641 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.181645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.182837 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.182839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.184145 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.184148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.185109 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.185110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.185209 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.185232 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:59:44.185344 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:59:44.185432 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":182,"threads_started":1}
I20260619 11:59:44.186081 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.181241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44757 (local address 127.0.0.1:41276)
0619 11:59:44.181551 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:44.181563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:44.181613 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:44.181953 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:44.181955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:44.181957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:44.181997 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.181998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.183455 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.183457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.184581 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.184584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.185791 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.185793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.185871 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.185879 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:59:44.185915 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 11:59:44.185975 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":80,"threads_started":1}
I20260619 11:59:44.186386 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.181193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.0.203.193:39383)
0619 11:59:44.181448 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.181455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.181466 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:59:44.181511 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.181513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.181533 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.181633 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:44.181844 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.183261 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.184705 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.185830 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.185901 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.185972 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.186208 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.186300 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":135,"threads_started":1}
I20260619 11:59:44.186448 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 35f425d17af24608ab8aac89bbd79b8b (DEFAULT_TABLE table=test-table [id=768ab4c909404a169c43a8f59e932001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:44.186610 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.181466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.0.203.195:44757)
0619 11:59:44.181765 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.181770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.181779 (+ 9us) server_negotiation.cc:407] Connection header received
0619 11:59:44.181819 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.181821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.181838 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.181932 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:44.182109 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.183353 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.184747 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.185665 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.186337 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.186409 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.186441 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.186502 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
I20260619 11:59:44.186785 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.181012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:39383 (local address 127.0.0.1:56252)
0619 11:59:44.181200 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:44.181217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:44.181305 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:44.181717 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:44.181719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:44.181722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:44.181762 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.181763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.183413 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.183415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.184554 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.184557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.186009 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.186010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.186078 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.186082 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:59:44.186149 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:59:44.186315 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260619 11:59:44.186908 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f425d17af24608ab8aac89bbd79b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:44.187620 1510 tablet_service.cc:1467] Processing CreateTablet for tablet 35f425d17af24608ab8aac89bbd79b8b (DEFAULT_TABLE table=test-table [id=768ab4c909404a169c43a8f59e932001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:44.187664 1778 tablet_service.cc:1467] Processing CreateTablet for tablet 35f425d17af24608ab8aac89bbd79b8b (DEFAULT_TABLE table=test-table [id=768ab4c909404a169c43a8f59e932001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:44.188086 1778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f425d17af24608ab8aac89bbd79b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:44.188095 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f425d17af24608ab8aac89bbd79b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:44.190078 1861 tablet_bootstrap.cc:492] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2: Bootstrap starting.
I20260619 11:59:44.190820 1861 tablet_bootstrap.cc:654] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:44.191362 1861 log.cc:824] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:44.192225 1861 tablet_bootstrap.cc:492] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2: No bootstrap required, opened a new log
I20260619 11:59:44.192410 1863 tablet_bootstrap.cc:492] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4: Bootstrap starting.
I20260619 11:59:44.192675 1861 ts_tablet_manager.cc:1397] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:59:44.193228 1863 tablet_bootstrap.cc:654] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:44.193840 1863 log.cc:824] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:44.194231 1864 tablet_bootstrap.cc:492] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042: Bootstrap starting.
I20260619 11:59:44.195439 1864 tablet_bootstrap.cc:654] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:44.195358 1861 raft_consensus.cc:348] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.195595 1861 raft_consensus.cc:374] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:44.195657 1861 raft_consensus.cc:729] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e7a824297d42719ce5516cc16c0bb2, State: Initialized, Role: FOLLOWER
I20260619 11:59:44.195967 1861 consensus_queue.cc:260] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.196233 1864 log.cc:824] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:44.196417 1861 ts_tablet_manager.cc:1428] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 11:59:44.196669 1710 heartbeater.cc:503] Master 127.0.203.254:33231 was elected leader, sending a full tablet report...
I20260619 11:59:44.197124 1863 tablet_bootstrap.cc:492] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4: No bootstrap required, opened a new log
I20260619 11:59:44.197214 1863 ts_tablet_manager.cc:1397] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 11:59:44.197259 1864 tablet_bootstrap.cc:492] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042: No bootstrap required, opened a new log
I20260619 11:59:44.197330 1864 ts_tablet_manager.cc:1397] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:59:44.199362 1864 raft_consensus.cc:348] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.199362 1863 raft_consensus.cc:348] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.199581 1864 raft_consensus.cc:374] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:44.199581 1863 raft_consensus.cc:374] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:44.199620 1864 raft_consensus.cc:729] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749a1436d010431fa433ab4d5fda6042, State: Initialized, Role: FOLLOWER
I20260619 11:59:44.199620 1863 raft_consensus.cc:729] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa70d62ee584c499c8aaeb41a2b5aa4, State: Initialized, Role: FOLLOWER
I20260619 11:59:44.199755 1864 consensus_queue.cc:260] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.199817 1863 consensus_queue.cc:260] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.200105 1864 ts_tablet_manager.cc:1428] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 11:59:44.200105 1863 ts_tablet_manager.cc:1428] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 11:59:44.200174 1844 heartbeater.cc:503] Master 127.0.203.254:33231 was elected leader, sending a full tablet report...
I20260619 11:59:44.200371 1576 heartbeater.cc:503] Master 127.0.203.254:33231 was elected leader, sending a full tablet report...
W20260619 11:59:44.225701 1577 tablet.cc:2369] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:59:44.293411 1866 raft_consensus.cc:482] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:44.293562 1866 raft_consensus.cc:504] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.293952 1866 leader_election.cc:288] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 749a1436d010431fa433ab4d5fda6042 (127.0.203.193:39383), 4fa70d62ee584c499c8aaeb41a2b5aa4 (127.0.203.195:44757)
I20260619 11:59:44.297811 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.294437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:33967 (local address 127.0.203.193:39383)
0619 11:59:44.294505 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.294508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.294526 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:59:44.294622 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.294624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.294645 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.294757 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:44.294956 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.295580 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.296899 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.297341 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.297458 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.297471 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.297660 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.297706 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:59:44.297811 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.294282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:39383 (local address 127.0.203.194:33967)
0619 11:59:44.294363 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:44.294381 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:44.294512 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:44.294764 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:44.294766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:44.294768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:44.294870 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.294872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.295708 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.295709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.296797 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.296800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.297458 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.297459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.297556 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.297560 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:59:44.297650 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:59:44.297727 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:59:44.298203 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.294552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:55603 (local address 127.0.203.195:44757)
0619 11:59:44.294606 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.294608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.294751 (+ 143us) server_negotiation.cc:407] Connection header received
0619 11:59:44.294829 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.294830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.294840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.294987 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:44.295182 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.295766 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.297223 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.297761 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.297850 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.297862 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.298047 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.298124 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 11:59:44.298209 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.294425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44757 (local address 127.0.203.194:55603)
0619 11:59:44.294676 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:44.294685 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:44.294765 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:44.294997 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:44.294998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:44.295000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:44.295097 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.295100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.295869 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.295871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.297116 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:44.297120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.297855 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.297856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:44.297945 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.297951 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:59:44.298042 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 11:59:44.298104 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":170,"threads_started":1}
I20260619 11:59:44.298277 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f425d17af24608ab8aac89bbd79b8b" candidate_uuid: "f5e7a824297d42719ce5516cc16c0bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749a1436d010431fa433ab4d5fda6042" is_pre_election: true
I20260619 11:59:44.298511 1530 raft_consensus.cc:2454] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e7a824297d42719ce5516cc16c0bb2 in term 0.
I20260619 11:59:44.298815 1600 leader_election.cc:302] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749a1436d010431fa433ab4d5fda6042, f5e7a824297d42719ce5516cc16c0bb2; no voters:
I20260619 11:59:44.298970 1866 raft_consensus.cc:2781] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:44.299067 1866 raft_consensus.cc:482] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:44.298995 1798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f425d17af24608ab8aac89bbd79b8b" candidate_uuid: "f5e7a824297d42719ce5516cc16c0bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" is_pre_election: true
I20260619 11:59:44.299100 1866 raft_consensus.cc:3037] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:44.299183 1798 raft_consensus.cc:2454] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e7a824297d42719ce5516cc16c0bb2 in term 0.
I20260619 11:59:44.300933 1866 raft_consensus.cc:504] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.301126 1866 leader_election.cc:288] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 749a1436d010431fa433ab4d5fda6042 (127.0.203.193:39383), 4fa70d62ee584c499c8aaeb41a2b5aa4 (127.0.203.195:44757)
I20260619 11:59:44.301390 1798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f425d17af24608ab8aac89bbd79b8b" candidate_uuid: "f5e7a824297d42719ce5516cc16c0bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4"
I20260619 11:59:44.301422 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f425d17af24608ab8aac89bbd79b8b" candidate_uuid: "f5e7a824297d42719ce5516cc16c0bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749a1436d010431fa433ab4d5fda6042"
I20260619 11:59:44.301497 1798 raft_consensus.cc:3037] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:44.301540 1530 raft_consensus.cc:3037] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:44.302477 1798 raft_consensus.cc:2454] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e7a824297d42719ce5516cc16c0bb2 in term 1.
I20260619 11:59:44.302450 1530 raft_consensus.cc:2454] T 35f425d17af24608ab8aac89bbd79b8b P 749a1436d010431fa433ab4d5fda6042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e7a824297d42719ce5516cc16c0bb2 in term 1.
I20260619 11:59:44.302698 1600 leader_election.cc:302] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fa70d62ee584c499c8aaeb41a2b5aa4, f5e7a824297d42719ce5516cc16c0bb2; no voters:
I20260619 11:59:44.302826 1866 raft_consensus.cc:2781] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:44.303040 1866 raft_consensus.cc:686] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [term 1 LEADER]: Becoming Leader. State: Replica: f5e7a824297d42719ce5516cc16c0bb2, State: Running, Role: LEADER
I20260619 11:59:44.303177 1866 consensus_queue.cc:237] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } }
I20260619 11:59:44.304271 1384 catalog_manager.cc:5495] T 35f425d17af24608ab8aac89bbd79b8b P f5e7a824297d42719ce5516cc16c0bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5e7a824297d42719ce5516cc16c0bb2 (127.0.203.194). New cstate: current_term: 1 leader_uuid: "f5e7a824297d42719ce5516cc16c0bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e7a824297d42719ce5516cc16c0bb2" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "749a1436d010431fa433ab4d5fda6042" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fa70d62ee584c499c8aaeb41a2b5aa4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44757 } health_report { overall_health: UNKNOWN } } }
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 11:59:44.342971 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.333673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.203.193:39383)
0619 11:59:44.333802 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.333805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.333935 (+ 130us) server_negotiation.cc:407] Connection header received
0619 11:59:44.334012 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.334014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.334027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.334128 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:44.334727 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.335221 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.336043 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.336242 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.336341 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.339182 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:44.339197 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:44.339197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:44.339211 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:44.341500 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.341975 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.341978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.341979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.342061 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.342323 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.342324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.342325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.342493 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:44.342672 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.342745 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.342843 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870384, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870384, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 11:59:44.373042 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.364005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.0.203.193:39383)
0619 11:59:44.364108 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.364111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.364193 (+ 82us) server_negotiation.cc:407] Connection header received
0619 11:59:44.364300 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.364302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.364314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.364376 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:44.365157 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.365716 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.366528 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.366744 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.366812 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.369645 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:44.369647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:44.369647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:44.369656 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:44.371374 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.371808 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.371809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.371809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.371882 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.372135 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.372136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.372136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.372270 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:44.372378 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.372780 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.372902 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260619 11:59:44.390453 1845 tablet.cc:2369] T 35f425d17af24608ab8aac89bbd79b8b P 4fa70d62ee584c499c8aaeb41a2b5aa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 11:59:44.400249 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:44.391024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.0.203.193:39383)
0619 11:59:44.391121 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:44.391123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:44.391204 (+ 81us) server_negotiation.cc:407] Connection header received
0619 11:59:44.391281 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:44.391283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:44.391294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:44.391357 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:44.392067 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.392595 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.393612 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:44.393834 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:44.393916 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:44.397040 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:44.397043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:44.397043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:44.397057 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:44.398780 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.399339 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.399341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.399341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.399405 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:44.399691 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:44.399691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:44.399691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:44.399807 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:44.399896 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:44.400078 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 11:59:44.400154 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 11:59:44.400919 1490 tablet_service.cc:2908] Rejecting scan request for tablet 35f425d17af24608ab8aac89bbd79b8b: Uninitialized: clean time has not yet been initialized
I20260619 11:59:44.402060 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 1447
I20260619 11:59:44.408046 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 1581
I20260619 11:59:44.413949 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 1715
I20260619 11:59:44.419595 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 1353
2026-06-19T11:59:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2280 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1885](info): set up 2 sockets
Jun 19 11:59:44 dist-test-slave-fbq8 krb5kdc[1885](info): commencing operation
krb5kdc: starting...
W20260619 11:59:45.504274 815 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:45 dist-test-slave-fbq8 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:45.644621 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:46117
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:45.739866 1898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:45.740187 1898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:45.740227 1898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:45.741465 1898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:45.741554 1898 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:59:45.741578 1898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:45.741609 1898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:45.741638 1898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:45.741660 1898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:45.743213 1898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:46117
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:45.743561 1898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:45.743834 1898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:45.749588 1904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:45.749624 1906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:45.749666 1898 server_base.cc:1034] running on GCE node
W20260619 11:59:45.749614 1903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:45.750218 1898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:45.750612 1898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:45.751850 1898 hybrid_clock.cc:648] HybridClock initialized: now 1781870385751831 us; error 45 us; skew 500 ppm
Jun 19 11:59:45 dist-test-slave-fbq8 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:45.760834 1898 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:45.761422 1898 webserver.cc:460] Webserver started at http://127.0.203.254:33459/ using document root <none> and password file <none>
I20260619 11:59:45.761696 1898 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:45.761776 1898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:45.761942 1898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:45.763137 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "91efef3053c64f48ad5c7f18facc10e5"
format_stamp: "Formatted at 2026-06-19 11:59:45 on dist-test-slave-fbq8"
server_key: "27fa555e5b0d8a3d1e9358f639b84514"
server_key_iv: "9c2cf978fd0b2c9922636098c4f55f95"
server_key_version: "encryptionkey@0"
I20260619 11:59:45.763561 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "91efef3053c64f48ad5c7f18facc10e5"
format_stamp: "Formatted at 2026-06-19 11:59:45 on dist-test-slave-fbq8"
server_key: "27fa555e5b0d8a3d1e9358f639b84514"
server_key_iv: "9c2cf978fd0b2c9922636098c4f55f95"
server_key_version: "encryptionkey@0"
I20260619 11:59:45.765493 1898 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:59:45.766525 1913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:45.766875 1898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:45.767016 1898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "91efef3053c64f48ad5c7f18facc10e5"
format_stamp: "Formatted at 2026-06-19 11:59:45 on dist-test-slave-fbq8"
server_key: "27fa555e5b0d8a3d1e9358f639b84514"
server_key_iv: "9c2cf978fd0b2c9922636098c4f55f95"
server_key_version: "encryptionkey@0"
I20260619 11:59:45.767115 1898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:45.789731 1898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:45.793239 1898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:45.793416 1898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:45.799953 1898 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:46117
I20260619 11:59:45.800035 1975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:46117 every 8 connection(s)
I20260619 11:59:45.800457 1898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:45.801470 1976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:45.801492 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 1898
I20260619 11:59:45.801579 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:45.801772 815 external_mini_cluster.cc:1428] Setting key 0dd07f747127a01734b972dc13926f3e
I20260619 11:59:45.805110 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: Bootstrap starting.
I20260619 11:59:45.805907 1976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:45.806237 1976 log.cc:824] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:45.807159 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: No bootstrap required, opened a new log
Jun 19 11:59:45 dist-test-slave-fbq8 krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:45.811049 1976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91efef3053c64f48ad5c7f18facc10e5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46117 } }
I20260619 11:59:45.811326 1976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:45.811381 1976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91efef3053c64f48ad5c7f18facc10e5, State: Initialized, Role: FOLLOWER
I20260619 11:59:45.811551 1976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91efef3053c64f48ad5c7f18facc10e5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46117 } }
I20260619 11:59:45.811674 1976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:45.811725 1976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:45.811779 1976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:45.813265 1976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91efef3053c64f48ad5c7f18facc10e5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46117 } }
I20260619 11:59:45.813455 1976 leader_election.cc:302] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91efef3053c64f48ad5c7f18facc10e5; no voters:
I20260619 11:59:45.813730 1976 leader_election.cc:288] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:45.814028 1982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:45.814054 1976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:45.814308 1982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [term 1 LEADER]: Becoming Leader. State: Replica: 91efef3053c64f48ad5c7f18facc10e5, State: Running, Role: LEADER
I20260619 11:59:45.814447 1982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91efef3053c64f48ad5c7f18facc10e5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46117 } }
I20260619 11:59:45.816201 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:45.802420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.0.203.254:46117)
0619 11:59:45.802625 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:45.802637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:45.802672 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:59:45.803223 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:45.803233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:45.803303 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:45.803473 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:45.804232 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:45.805680 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:45.806650 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:45.807136 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:45.807236 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:45.810785 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:45.810791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:45.810792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:45.810832 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:45.813575 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:45.814169 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:45.814173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:45.814174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:45.814316 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:45.814594 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:45.814595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:45.814596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:45.814966 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:45.815129 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:45.815802 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 11:59:45.815957 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
W20260619 11:59:45.816808 1996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:59:45.816864 1996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:59:45.816385 1981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91efef3053c64f48ad5c7f18facc10e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91efef3053c64f48ad5c7f18facc10e5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46117 } } }
I20260619 11:59:45.816473 1983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91efef3053c64f48ad5c7f18facc10e5. Latest consensus state: current_term: 1 leader_uuid: "91efef3053c64f48ad5c7f18facc10e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91efef3053c64f48ad5c7f18facc10e5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46117 } } }
I20260619 11:59:45.817130 1983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:45.817320 1981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5 [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:45.817430 1998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:45.817696 1998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:59:45.819684 1998 catalog_manager.cc:1347] Generated new cluster ID: 8ceb9d460671459f8d21d91f95c55f3b
I20260619 11:59:45.819792 1998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:59:45.842665 1998 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:45.843585 1998 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:45.850011 1998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: Generated new TSK 0
I20260619 11:59:45.850260 1998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 11:59:45.888310 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46117
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:45.987907 2002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:45.988394 2002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:45.988456 2002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:45.989957 2002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:45.990018 2002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:45.990042 2002 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:59:45.990063 2002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 11:59:45.991771 2002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:45.992089 2002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:45.992523 2002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:45.998677 2010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:45.998677 2008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:45.998824 2007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:46.000232 2002 server_base.cc:1034] running on GCE node
I20260619 11:59:46.000905 2002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:46.001274 2002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:46.002480 2002 hybrid_clock.cc:648] HybridClock initialized: now 1781870386002469 us; error 34 us; skew 500 ppm
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:46.005625 2002 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 11:59:46.006132 2002 webserver.cc:460] Webserver started at http://127.0.203.193:40613/ using document root <none> and password file <none>
I20260619 11:59:46.006413 2002 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:46.006496 2002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:46.006676 2002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:46.008064 2002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "210cf01a7431498ba516ad475acb3e90"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "cab263a81c3658d57c3af922dded921d"
server_key_iv: "ec73174faff0190bb790306f82777314"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.008526 2002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "210cf01a7431498ba516ad475acb3e90"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "cab263a81c3658d57c3af922dded921d"
server_key_iv: "ec73174faff0190bb790306f82777314"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.010332 2002 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 11:59:46.011621 2017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.012013 2002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:46.012156 2002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "210cf01a7431498ba516ad475acb3e90"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "cab263a81c3658d57c3af922dded921d"
server_key_iv: "ec73174faff0190bb790306f82777314"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.012262 2002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:46.027506 2002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:46.031385 2002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:46.031539 2002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:46.032132 2002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:46.032173 2002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.032198 2002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:46.032212 2002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:46.042372 2002 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:33883
I20260619 11:59:46.042490 2132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:33883 every 8 connection(s)
I20260619 11:59:46.042919 2002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 11:59:46.045217 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 2002
I20260619 11:59:46.045364 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 11:59:46.045536 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.033083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:59665 (local address 127.0.203.254:46117)
0619 11:59:46.033177 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.033180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.033508 (+ 328us) server_negotiation.cc:407] Connection header received
0619 11:59:46.034496 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.034497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.034512 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.034615 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.035698 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.036555 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.037752 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.037993 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.038079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.041915 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.041917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.041918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.041929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.043735 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.044248 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.044249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.044249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.044321 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.044741 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.044743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.044743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.044898 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.044989 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.045318 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.045437 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:59:46.045632 815 external_mini_cluster.cc:1428] Setting key e0984982361c72ff5610d308f7c7b837
I20260619 11:59:46.045756 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.033043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46117 (local address 127.0.203.193:59665)
0619 11:59:46.033300 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.033365 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.034357 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.034716 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.034722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.035184 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:46.035571 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.035577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.036698 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.036700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.037613 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.037617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.038101 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.038102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.038348 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.039165 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:46.039181 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:46.041772 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:46.043843 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.043848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.043850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.044115 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.044450 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.044452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.044452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.044592 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.045004 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:46.045010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:46.045167 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.045285 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.045490 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":125,"threads_started":1}
I20260619 11:59:46.046418 2133 heartbeater.cc:347] Connected to a master server at 127.0.203.254:46117
I20260619 11:59:46.046509 2133 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:46.046716 2133 heartbeater.cc:511] Master 127.0.203.254:46117 requested a full tablet report, sending...
I20260619 11:59:46.047291 1930 ts_manager.cc:194] Registered new tserver with Master: 210cf01a7431498ba516ad475acb3e90 (127.0.203.193:33883)
I20260619 11:59:46.048224 1930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:59665
I20260619 11:59:46.056299 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.047673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:45367 (local address 127.0.203.254:46117)
0619 11:59:46.047725 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.047727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.047830 (+ 103us) server_negotiation.cc:407] Connection header received
0619 11:59:46.047890 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.047891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.047905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.047987 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.048392 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.049066 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.049842 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.050063 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.050121 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.051638 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.051640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.051640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.051651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.054202 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.054783 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.054784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.054784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.054860 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.055532 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.055534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.055534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.055692 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.055778 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.056096 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.056218 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 11:59:46.056336 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.047587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46117 (local address 127.0.203.193:45367)
0619 11:59:46.047749 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.047760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.047820 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.048009 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.048009 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.048214 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:46.048282 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.048285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.049166 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.049166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.049724 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.049725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.050149 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.050150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.050335 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.050970 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:46.050977 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:46.051449 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:46.054348 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.054350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.054351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.054631 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.054965 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.054966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.054966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.055048 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.055829 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:46.055831 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:46.055901 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.056011 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.056153 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 11:59:46.082252 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46117
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:46.169087 2142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:46.169322 2142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:46.169349 2142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:46.170682 2142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:46.170740 2142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:46.170764 2142 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:59:46.170783 2142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 11:59:46.172426 2142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:46.172724 2142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:46.172963 2142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:59:46.178761 2142 server_base.cc:1034] running on GCE node
W20260619 11:59:46.178786 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:46.178778 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:46.178982 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:46.179345 2142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:46.179594 2142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:46.180745 2142 hybrid_clock.cc:648] HybridClock initialized: now 1781870386180671 us; error 95 us; skew 500 ppm
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:46.183939 2142 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 11:59:46.184477 2142 webserver.cc:460] Webserver started at http://127.0.203.194:34233/ using document root <none> and password file <none>
I20260619 11:59:46.184857 2142 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:46.184942 2142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:46.185115 2142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:46.186290 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "0d59019c95c34d36a5dff384cabd8864"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "193ffdfc97375818f674d20b8e3a5804"
server_key_iv: "963a6c8922e4ced6d97a2b44258a8c52"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.186702 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "0d59019c95c34d36a5dff384cabd8864"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "193ffdfc97375818f674d20b8e3a5804"
server_key_iv: "963a6c8922e4ced6d97a2b44258a8c52"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.188369 2142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 11:59:46.189566 2157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.189944 2142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:46.190083 2142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "0d59019c95c34d36a5dff384cabd8864"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "193ffdfc97375818f674d20b8e3a5804"
server_key_iv: "963a6c8922e4ced6d97a2b44258a8c52"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.190188 2142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:46.204175 2142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:46.206689 2142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:46.206882 2142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:46.207444 2142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:46.207511 2142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.207554 2142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:46.207587 2142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:46.215768 2142 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:40889
I20260619 11:59:46.216207 2142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 11:59:46.216747 2272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:40889 every 8 connection(s)
I20260619 11:59:46.218814 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 2142
I20260619 11:59:46.218958 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 11:59:46.219165 815 external_mini_cluster.cc:1428] Setting key 3315d7d6bd1d7232dc5ef821a410722e
I20260619 11:59:46.220005 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.208226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:50461 (local address 127.0.203.254:46117)
0619 11:59:46.208283 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.208306 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.208595 (+ 289us) server_negotiation.cc:407] Connection header received
0619 11:59:46.209458 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.209459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.209473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.209538 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.210577 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.211531 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.213080 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.213310 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.213409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.216542 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.216545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.216545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.216555 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.218260 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.218756 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.218758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.218759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.218849 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.219115 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.219116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.219117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.219285 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.219363 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.219790 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.219893 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 11:59:46.220580 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.208188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46117 (local address 127.0.203.194:50461)
0619 11:59:46.208383 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.208457 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.209329 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.209627 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.209632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.210103 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:46.210480 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.210486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.211615 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.211617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.212982 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.212985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.213418 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.213419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.213571 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.214264 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:46.214277 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:46.216369 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:46.218363 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.218369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.218371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.218625 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.218927 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.218928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.218928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.219061 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.219382 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:46.219390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:46.219625 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.219750 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.219933 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260619 11:59:46.221403 2273 heartbeater.cc:347] Connected to a master server at 127.0.203.254:46117
I20260619 11:59:46.221534 2273 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:46.221796 2273 heartbeater.cc:511] Master 127.0.203.254:46117 requested a full tablet report, sending...
I20260619 11:59:46.222298 1930 ts_manager.cc:194] Registered new tserver with Master: 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194:40889)
I20260619 11:59:46.222821 1930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:50461
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
I20260619 11:59:46.230544 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.222972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:51887 (local address 127.0.203.254:46117)
0619 11:59:46.223033 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.223035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.223047 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:59:46.223114 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.223115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.223129 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.223198 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.223644 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.224293 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.225314 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.225536 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.225635 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.226995 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.226997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.226997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.227005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.229141 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.229548 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.229549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.229549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.229611 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.229827 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.229828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.229828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.229995 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.230078 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.230247 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.230336 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 11:59:46.230576 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.222742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46117 (local address 127.0.203.194:51887)
0619 11:59:46.222930 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.222948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.223037 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.223205 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.223206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.223449 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:46.223530 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.223532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.224453 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.224454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.225191 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.225194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.225639 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.225640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.225712 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.226397 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:46.226402 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:46.226841 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:46.229252 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.229254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.229254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.229450 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.229688 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.229689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.229690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.229754 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.230083 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:46.230084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:46.230142 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.230222 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.230338 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 11:59:46.253901 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46117
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:46.349762 2282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:46.350000 2282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:46.350031 2282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:46.351312 2282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:46.351356 2282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:46.351378 2282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:59:46.351397 2282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 11:59:46.353042 2282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:46.353313 2282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:46.353592 2282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:46.359350 2287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:46.359367 2288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:46.359350 2290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:46.359802 2282 server_base.cc:1034] running on GCE node
I20260619 11:59:46.360103 2282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:46.360420 2282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:46.361614 2282 hybrid_clock.cc:648] HybridClock initialized: now 1781870386361587 us; error 48 us; skew 500 ppm
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:46.364837 2282 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 11:59:46.365407 2282 webserver.cc:460] Webserver started at http://127.0.203.195:41731/ using document root <none> and password file <none>
I20260619 11:59:46.365681 2282 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:46.365765 2282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:46.365942 2282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:46.367182 2282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "fc7f425af7cb4538ba206b8145e07d55"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "17edeb00c2bfb598fe9c3c9e17891489"
server_key_iv: "95a6bfca4d6295455815a82fe29cbad7"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.367607 2282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "fc7f425af7cb4538ba206b8145e07d55"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "17edeb00c2bfb598fe9c3c9e17891489"
server_key_iv: "95a6bfca4d6295455815a82fe29cbad7"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.369324 2282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 11:59:46.370374 2297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.370631 2282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:46.370756 2282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "fc7f425af7cb4538ba206b8145e07d55"
format_stamp: "Formatted at 2026-06-19 11:59:46 on dist-test-slave-fbq8"
server_key: "17edeb00c2bfb598fe9c3c9e17891489"
server_key_iv: "95a6bfca4d6295455815a82fe29cbad7"
server_key_version: "encryptionkey@0"
I20260619 11:59:46.370864 2282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:46.384028 2282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:46.386583 2282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:46.386778 2282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:46.387399 2282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:46.387472 2282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.387516 2282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:46.387573 2282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:46.396260 2282 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:39671
I20260619 11:59:46.396474 2412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:39671 every 8 connection(s)
I20260619 11:59:46.396785 2282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 11:59:46.400041 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.388381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:53537 (local address 127.0.203.254:46117)
0619 11:59:46.388466 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.388470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.388657 (+ 187us) server_negotiation.cc:407] Connection header received
0619 11:59:46.389344 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.389346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.389360 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.389457 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.390600 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.391472 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.393297 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.393635 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.393723 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.396906 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.396908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.396908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.396917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.398504 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.398980 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.398981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.398982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.399045 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.399323 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.399324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.399324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.399453 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.399536 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.399842 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.399930 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:59:46.400307 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.388162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46117 (local address 127.0.203.195:53537)
0619 11:59:46.388463 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.388519 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.389224 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.389636 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.389642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.390131 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:46.390453 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.390462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.391625 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.391626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.393175 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.393179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.393770 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.393771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.393951 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.394557 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:46.394570 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:46.396784 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:46.398627 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.398634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.398637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.398891 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.399131 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.399132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.399132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.399250 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.399525 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:46.399532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:46.399696 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.399813 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.400009 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260619 11:59:46.400348 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 2282
I20260619 11:59:46.400465 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 11:59:46.400686 815 external_mini_cluster.cc:1428] Setting key 3dc7c12ae8959fb2d4b616b43da33ea3
I20260619 11:59:46.402081 2413 heartbeater.cc:347] Connected to a master server at 127.0.203.254:46117
I20260619 11:59:46.402179 2413 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:46.402381 2413 heartbeater.cc:511] Master 127.0.203.254:46117 requested a full tablet report, sending...
I20260619 11:59:46.402794 1929 ts_manager.cc:194] Registered new tserver with Master: fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195:39671)
I20260619 11:59:46.403308 1929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:53537
I20260619 11:59:46.409148 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.402206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46117 (local address 127.0.203.195:48771)
0619 11:59:46.402375 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.402390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.402472 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.402648 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.402649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.402907 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:46.403003 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.403005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.403944 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.403945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.404540 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.404541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.405073 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.405074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.405135 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.405660 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:46.405665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:46.406033 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:46.407849 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.407850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.407851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.408100 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.408389 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.408390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.408390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.408455 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.408850 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:46.408851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:46.408900 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.408978 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.409070 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260619 11:59:46.409230 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.402348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:48771 (local address 127.0.203.254:46117)
0619 11:59:46.402410 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.402413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.402448 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:59:46.402547 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.402547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.402558 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.402616 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.403122 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.403805 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.404710 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.404940 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.405008 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.406154 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.406155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.406155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.406161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.407716 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.408225 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.408226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.408226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.408285 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.408566 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.408567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.408567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.408770 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.408835 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.409055 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.409122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 11:59:46.413122 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:46.452939 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.443837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.0.203.254:46117)
0619 11:59:46.443903 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.443906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.443919 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:59:46.443959 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.443960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.443974 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.444082 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.445075 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.445932 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.447163 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.447417 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.447512 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.450180 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.450182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.450182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.450189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.451615 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.452010 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.452010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.452011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.452060 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.452280 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.452281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.452282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.452421 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.452515 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.452725 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.452792 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:59:46.455070 1929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:59:46.457247 1929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:46.465222 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.460971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.0.203.194:40889)
0619 11:59:46.461503 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.461507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.461525 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:59:46.461556 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.461557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.461568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.461670 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:46.461929 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.462824 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.464126 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.464710 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.464816 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.464906 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.465052 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.465100 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":122,"threads_started":1}
I20260619 11:59:46.465214 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.460745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40889 (local address 127.0.0.1:42472)
0619 11:59:46.461100 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.461134 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.461203 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.461755 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.461757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.461760 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:46.461836 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.461838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.462931 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.462932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.464047 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.464049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.464819 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.464820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.464908 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.464925 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.465035 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.465101 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":134,"threads_started":1}
I20260619 11:59:46.465411 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.460754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.0.203.193:33883)
0619 11:59:46.460971 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.460978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.461250 (+ 272us) server_negotiation.cc:407] Connection header received
0619 11:59:46.461292 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.461294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.461310 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.461391 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:46.461655 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.462456 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.464081 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.464871 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.464945 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.465021 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.465232 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.465273 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260619 11:59:46.465900 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.460608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:33883 (local address 127.0.0.1:44192)
0619 11:59:46.461045 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.461086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.461162 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.461452 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.461457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.461460 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:46.461551 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.461553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.462554 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.462555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.463982 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.463984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.465061 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.465062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.465136 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.465141 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.465747 (+ 606us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.465813 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":30,"thread_start_us":116,"threads_started":1}
I20260619 11:59:46.466463 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.460922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:39671 (local address 127.0.0.1:40854)
0619 11:59:46.461185 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.461196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.461249 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.462748 (+ 1499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.462749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.462752 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:46.462818 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.462820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.463811 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.463812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.464983 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.464986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.466216 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.466216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.466290 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.466294 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.466349 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.466393 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260619 11:59:46.466364 2207 tablet_service.cc:1467] Processing CreateTablet for tablet a1e34a256af34d41904c45cfdd58c755 (DEFAULT_TABLE table=test-table [id=c708ca23e95b4590b0516cf04115b974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:46.466742 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.461491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.0.203.195:39671)
0619 11:59:46.462591 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.462594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.462603 (+ 9us) server_negotiation.cc:407] Connection header received
0619 11:59:46.462632 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.462634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.462646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.462748 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:46.462874 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.463716 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.465072 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.466145 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.466205 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.466269 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.466620 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.466658 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":121,"threads_started":1}
I20260619 11:59:46.466813 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e34a256af34d41904c45cfdd58c755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.466861 2067 tablet_service.cc:1467] Processing CreateTablet for tablet a1e34a256af34d41904c45cfdd58c755 (DEFAULT_TABLE table=test-table [id=c708ca23e95b4590b0516cf04115b974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:46.467286 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e34a256af34d41904c45cfdd58c755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.467653 2347 tablet_service.cc:1467] Processing CreateTablet for tablet a1e34a256af34d41904c45cfdd58c755 (DEFAULT_TABLE table=test-table [id=c708ca23e95b4590b0516cf04115b974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:46.467950 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e34a256af34d41904c45cfdd58c755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.470752 2436 tablet_bootstrap.cc:492] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Bootstrap starting.
I20260619 11:59:46.471417 2436 tablet_bootstrap.cc:654] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.471516 2438 tablet_bootstrap.cc:492] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Bootstrap starting.
I20260619 11:59:46.472103 2436 log.cc:824] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:46.472256 2438 tablet_bootstrap.cc:654] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.472579 2438 log.cc:824] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:46.473106 2436 tablet_bootstrap.cc:492] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: No bootstrap required, opened a new log
I20260619 11:59:46.473191 2436 ts_tablet_manager.cc:1397] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:59:46.473541 2438 tablet_bootstrap.cc:492] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: No bootstrap required, opened a new log
I20260619 11:59:46.473655 2438 ts_tablet_manager.cc:1397] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:59:46.475553 2436 raft_consensus.cc:348] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.475761 2436 raft_consensus.cc:374] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.475800 2436 raft_consensus.cc:729] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59019c95c34d36a5dff384cabd8864, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.475940 2436 consensus_queue.cc:260] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.476009 2438 raft_consensus.cc:348] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.476166 2438 raft_consensus.cc:374] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.476199 2438 raft_consensus.cc:729] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7f425af7cb4538ba206b8145e07d55, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.476342 2438 consensus_queue.cc:260] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.476405 2437 tablet_bootstrap.cc:492] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Bootstrap starting.
I20260619 11:59:46.476511 2273 heartbeater.cc:503] Master 127.0.203.254:46117 was elected leader, sending a full tablet report...
I20260619 11:59:46.476830 2436 ts_tablet_manager.cc:1428] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 11:59:46.477826 2438 ts_tablet_manager.cc:1428] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 11:59:46.477867 2413 heartbeater.cc:503] Master 127.0.203.254:46117 was elected leader, sending a full tablet report...
I20260619 11:59:46.478353 2437 tablet_bootstrap.cc:654] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.478732 2437 log.cc:824] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:46.479504 2437 tablet_bootstrap.cc:492] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: No bootstrap required, opened a new log
I20260619 11:59:46.479566 2437 ts_tablet_manager.cc:1397] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:59:46.481138 2437 raft_consensus.cc:348] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.481298 2437 raft_consensus.cc:374] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.481333 2437 raft_consensus.cc:729] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210cf01a7431498ba516ad475acb3e90, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.481475 2437 consensus_queue.cc:260] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.481753 2437 ts_tablet_manager.cc:1428] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:59:46.481876 2133 heartbeater.cc:503] Master 127.0.203.254:46117 was elected leader, sending a full tablet report...
W20260619 11:59:46.544178 2134 tablet.cc:2369] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:59:46.570757 2442 raft_consensus.cc:482] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:46.570916 2442 raft_consensus.cc:504] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.571376 2442 leader_election.cc:288] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 210cf01a7431498ba516ad475acb3e90 (127.0.203.193:33883), 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194:40889)
I20260619 11:59:46.575325 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.571660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40889 (local address 127.0.203.195:58875)
0619 11:59:46.571769 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.571783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.571859 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.572093 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.572095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.572097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:46.572225 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.572228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.573073 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.573075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.574156 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.574158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.574995 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.574996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.575111 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.575116 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.575192 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.575237 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 11:59:46.575351 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.571767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:58875 (local address 127.0.203.194:40889)
0619 11:59:46.571828 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.571831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.571847 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:59:46.571979 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.571980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.571993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.572060 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:46.572345 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.572967 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.574251 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.574904 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.574990 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.575003 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.575202 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.575272 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 11:59:46.575809 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e34a256af34d41904c45cfdd58c755" candidate_uuid: "fc7f425af7cb4538ba206b8145e07d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59019c95c34d36a5dff384cabd8864" is_pre_election: true
I20260619 11:59:46.575899 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.571712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:33883 (local address 127.0.203.195:44015)
0619 11:59:46.571933 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.571945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.572031 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.572327 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.572328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.572332 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:46.572434 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.572436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.573517 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.573518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.574652 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.574655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.575624 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.575625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.575699 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.575704 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.575766 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.575819 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":129,"threads_started":1}
I20260619 11:59:46.575950 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.571797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:44015 (local address 127.0.203.193:33883)
0619 11:59:46.571847 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.571849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.572090 (+ 241us) server_negotiation.cc:407] Connection header received
0619 11:59:46.572120 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.572121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.572131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.572204 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:46.572581 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.573386 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.574796 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.575488 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.575558 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.575566 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.575806 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.575874 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 11:59:46.576035 2227 raft_consensus.cc:2454] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7f425af7cb4538ba206b8145e07d55 in term 0.
I20260619 11:59:46.576311 2300 leader_election.cc:302] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59019c95c34d36a5dff384cabd8864, fc7f425af7cb4538ba206b8145e07d55; no voters:
I20260619 11:59:46.576370 2087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e34a256af34d41904c45cfdd58c755" candidate_uuid: "fc7f425af7cb4538ba206b8145e07d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210cf01a7431498ba516ad475acb3e90" is_pre_election: true
I20260619 11:59:46.576478 2442 raft_consensus.cc:2781] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:46.576539 2442 raft_consensus.cc:482] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:46.576572 2442 raft_consensus.cc:3037] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:46.576562 2087 raft_consensus.cc:2454] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7f425af7cb4538ba206b8145e07d55 in term 0.
I20260619 11:59:46.577714 2442 raft_consensus.cc:504] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.577896 2442 leader_election.cc:288] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [CANDIDATE]: Term 1 election: Requested vote from peers 210cf01a7431498ba516ad475acb3e90 (127.0.203.193:33883), 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194:40889)
I20260619 11:59:46.578130 2087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e34a256af34d41904c45cfdd58c755" candidate_uuid: "fc7f425af7cb4538ba206b8145e07d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210cf01a7431498ba516ad475acb3e90"
I20260619 11:59:46.578109 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e34a256af34d41904c45cfdd58c755" candidate_uuid: "fc7f425af7cb4538ba206b8145e07d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59019c95c34d36a5dff384cabd8864"
I20260619 11:59:46.578256 2087 raft_consensus.cc:3037] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:46.578251 2227 raft_consensus.cc:3037] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:46.579047 2087 raft_consensus.cc:2454] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7f425af7cb4538ba206b8145e07d55 in term 1.
I20260619 11:59:46.579147 2227 raft_consensus.cc:2454] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7f425af7cb4538ba206b8145e07d55 in term 1.
I20260619 11:59:46.579427 2298 leader_election.cc:302] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210cf01a7431498ba516ad475acb3e90, fc7f425af7cb4538ba206b8145e07d55; no voters:
I20260619 11:59:46.579574 2442 raft_consensus.cc:2781] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:46.579772 2442 raft_consensus.cc:686] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 1 LEADER]: Becoming Leader. State: Replica: fc7f425af7cb4538ba206b8145e07d55, State: Running, Role: LEADER
I20260619 11:59:46.579959 2442 consensus_queue.cc:237] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.580924 1929 catalog_manager.cc:5495] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195). New cstate: current_term: 1 leader_uuid: "fc7f425af7cb4538ba206b8145e07d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } health_report { overall_health: HEALTHY } } }
I20260619 11:59:46.613348 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.610087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.0.203.195:39671)
0619 11:59:46.610182 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.610184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.610194 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:59:46.610265 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.610266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.610283 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.610356 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:46.610574 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.611115 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.612043 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.612233 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.612308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.612684 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:46.612792 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:46.612982 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.613207 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.613236 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:59:46.616094 2227 raft_consensus.cc:1261] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Refusing update from remote peer fc7f425af7cb4538ba206b8145e07d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:46.616102 2087 raft_consensus.cc:1261] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Refusing update from remote peer fc7f425af7cb4538ba206b8145e07d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:46.616636 2446 consensus_queue.cc:1035] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:46.616847 2442 consensus_queue.cc:1035] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:46.620604 2451 mvcc.cc:205] Tried to move back new op lower bound from 7298541103575179264 to 7298541103432314880. Current Snapshot: MvccSnapshot[applied={T|T < 7298541103575179264 or (T in {7298541103575179264})}]
I20260619 11:59:46.624281 1930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:46.624445 1930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:46.625471 1930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: Sending DeleteTablet for 3 replicas of tablet a1e34a256af34d41904c45cfdd58c755
I20260619 11:59:46.625914 2347 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e34a256af34d41904c45cfdd58c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:46 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:40854
I20260619 11:59:46.625972 2207 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e34a256af34d41904c45cfdd58c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:46 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:42472
I20260619 11:59:46.625989 2067 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e34a256af34d41904c45cfdd58c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:46 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44192
I20260619 11:59:46.626237 2459 tablet_replica.cc:330] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: stopping tablet replica
I20260619 11:59:46.626281 2460 tablet_replica.cc:330] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: stopping tablet replica
I20260619 11:59:46.626395 2459 raft_consensus.cc:2229] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:46.626397 2460 raft_consensus.cc:2229] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:46.626539 2460 raft_consensus.cc:2258] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:46.626539 2459 raft_consensus.cc:2258] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:46.626657 1930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:59:46.626848 1930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:46.627084 2460 ts_tablet_manager.cc:1905] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:46.627329 2458 tablet_replica.cc:330] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: stopping tablet replica
I20260619 11:59:46.627863 2459 ts_tablet_manager.cc:1905] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:46.629308 2067 tablet_service.cc:1467] Processing CreateTablet for tablet c047ace075b34786aa7de1ed99571845 (DEFAULT_TABLE table=test-table [id=a58accc8e48f49f58b15e3e226709828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:46.629608 2460 ts_tablet_manager.cc:1918] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:59:46.629689 2460 log.cc:1196] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/a1e34a256af34d41904c45cfdd58c755
I20260619 11:59:46.629755 2207 tablet_service.cc:1467] Processing CreateTablet for tablet c047ace075b34786aa7de1ed99571845 (DEFAULT_TABLE table=test-table [id=a58accc8e48f49f58b15e3e226709828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:46.629933 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047ace075b34786aa7de1ed99571845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.630009 2460 ts_tablet_manager.cc:1939] T a1e34a256af34d41904c45cfdd58c755 P 0d59019c95c34d36a5dff384cabd8864: Deleting consensus metadata
I20260619 11:59:46.630604 2459 ts_tablet_manager.cc:1918] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:59:46.630716 2459 log.cc:1196] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/a1e34a256af34d41904c45cfdd58c755
I20260619 11:59:46.631058 2459 ts_tablet_manager.cc:1939] T a1e34a256af34d41904c45cfdd58c755 P 210cf01a7431498ba516ad475acb3e90: Deleting consensus metadata
I20260619 11:59:46.631608 2436 tablet_bootstrap.cc:492] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Bootstrap starting.
I20260619 11:59:46.631707 2347 tablet_service.cc:1467] Processing CreateTablet for tablet c047ace075b34786aa7de1ed99571845 (DEFAULT_TABLE table=test-table [id=a58accc8e48f49f58b15e3e226709828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:46.631858 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047ace075b34786aa7de1ed99571845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.632153 2436 tablet_bootstrap.cc:654] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.632819 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047ace075b34786aa7de1ed99571845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.633209 2436 tablet_bootstrap.cc:492] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: No bootstrap required, opened a new log
I20260619 11:59:46.633253 2436 ts_tablet_manager.cc:1397] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 11:59:46.633440 2436 raft_consensus.cc:348] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.633533 2436 raft_consensus.cc:374] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.633563 2436 raft_consensus.cc:729] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59019c95c34d36a5dff384cabd8864, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.633626 2436 consensus_queue.cc:260] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.633745 2436 ts_tablet_manager.cc:1428] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 11:59:46.634284 2438 tablet_bootstrap.cc:492] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Bootstrap starting.
I20260619 11:59:46.634327 1916 catalog_manager.cc:4841] TS 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194:40889): tablet a1e34a256af34d41904c45cfdd58c755 (table test-table [id=c708ca23e95b4590b0516cf04115b974]) successfully deleted
I20260619 11:59:46.634371 2437 tablet_bootstrap.cc:492] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: Bootstrap starting.
I20260619 11:59:46.634799 2458 raft_consensus.cc:2229] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:59:46.634860 2437 tablet_bootstrap.cc:654] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.634832 2438 tablet_bootstrap.cc:654] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.635332 2458 raft_consensus.cc:2258] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:46.635552 1914 catalog_manager.cc:4841] TS 210cf01a7431498ba516ad475acb3e90 (127.0.203.193:33883): tablet a1e34a256af34d41904c45cfdd58c755 (table test-table [id=c708ca23e95b4590b0516cf04115b974]) successfully deleted
I20260619 11:59:46.635931 2458 ts_tablet_manager.cc:1905] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:46.635999 2438 tablet_bootstrap.cc:492] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: No bootstrap required, opened a new log
I20260619 11:59:46.636034 2438 ts_tablet_manager.cc:1397] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:59:46.636195 2438 raft_consensus.cc:348] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.636273 2438 raft_consensus.cc:374] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.636294 2438 raft_consensus.cc:729] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7f425af7cb4538ba206b8145e07d55, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.636349 2438 consensus_queue.cc:260] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.636361 2437 tablet_bootstrap.cc:492] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: No bootstrap required, opened a new log
I20260619 11:59:46.636406 2437 ts_tablet_manager.cc:1397] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:59:46.636479 2438 ts_tablet_manager.cc:1428] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.636617 2437 raft_consensus.cc:348] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.636754 2437 raft_consensus.cc:374] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.636787 2437 raft_consensus.cc:729] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210cf01a7431498ba516ad475acb3e90, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.636837 2437 consensus_queue.cc:260] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.636953 2437 ts_tablet_manager.cc:1428] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:46.638240 2458 ts_tablet_manager.cc:1918] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:59:46.638345 2458 log.cc:1196] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/a1e34a256af34d41904c45cfdd58c755
I20260619 11:59:46.638615 2458 ts_tablet_manager.cc:1939] T a1e34a256af34d41904c45cfdd58c755 P fc7f425af7cb4538ba206b8145e07d55: Deleting consensus metadata
I20260619 11:59:46.639139 1914 catalog_manager.cc:4841] TS fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195:39671): tablet a1e34a256af34d41904c45cfdd58c755 (table test-table [id=c708ca23e95b4590b0516cf04115b974]) successfully deleted
W20260619 11:59:46.647750 2414 tablet.cc:2369] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:59:46.717559 2274 tablet.cc:2369] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:59:46.728101 2444 raft_consensus.cc:482] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:46.728271 2444 raft_consensus.cc:504] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.728721 2444 leader_election.cc:288] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194:40889), fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195:39671)
I20260619 11:59:46.732897 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.728991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40889 (local address 127.0.203.193:34251)
0619 11:59:46.729305 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.729319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.729437 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.729646 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.729647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.729650 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:46.729715 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.729717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.730453 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.730454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.731885 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.731888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.732575 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.732576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.732699 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.732704 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.732754 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.732797 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":43,"thread_start_us":136,"threads_started":1}
I20260619 11:59:46.732878 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.729073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:34251 (local address 127.0.203.194:40889)
0619 11:59:46.729127 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.729130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.729397 (+ 267us) server_negotiation.cc:407] Connection header received
0619 11:59:46.729541 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.729542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.729553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.729613 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:46.729801 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.730364 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.732018 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.732461 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.732556 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.732569 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.732770 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.732810 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 11:59:46.733556 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.728959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:39671 (local address 127.0.203.193:53941)
0619 11:59:46.729129 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.729141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.729210 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.729405 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.729406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.729410 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:46.729532 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.729535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.730511 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.730512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.731885 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.731888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.733298 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.733299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.733356 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.733360 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.733448 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.733485 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20260619 11:59:46.733474 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047ace075b34786aa7de1ed99571845" candidate_uuid: "210cf01a7431498ba516ad475acb3e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59019c95c34d36a5dff384cabd8864" is_pre_election: true
I20260619 11:59:46.733623 2227 raft_consensus.cc:2454] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210cf01a7431498ba516ad475acb3e90 in term 0.
I20260619 11:59:46.733623 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.729204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:53941 (local address 127.0.203.195:39671)
0619 11:59:46.729250 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.729251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.729261 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:59:46.729288 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.729289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.729300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.729386 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:46.729648 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.730429 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.732353 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.733075 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.733165 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.733189 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.733467 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.733541 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 11:59:46.733877 2020 leader_election.cc:302] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59019c95c34d36a5dff384cabd8864, 210cf01a7431498ba516ad475acb3e90; no voters:
I20260619 11:59:46.733923 2367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047ace075b34786aa7de1ed99571845" candidate_uuid: "210cf01a7431498ba516ad475acb3e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7f425af7cb4538ba206b8145e07d55" is_pre_election: true
I20260619 11:59:46.734054 2367 raft_consensus.cc:2454] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 210cf01a7431498ba516ad475acb3e90 in term 0.
I20260619 11:59:46.734066 2444 raft_consensus.cc:2781] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:46.734117 2444 raft_consensus.cc:482] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:46.734141 2444 raft_consensus.cc:3037] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:46.734895 2444 raft_consensus.cc:504] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.735078 2444 leader_election.cc:288] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [CANDIDATE]: Term 1 election: Requested vote from peers 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194:40889), fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195:39671)
I20260619 11:59:46.735319 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047ace075b34786aa7de1ed99571845" candidate_uuid: "210cf01a7431498ba516ad475acb3e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59019c95c34d36a5dff384cabd8864"
I20260619 11:59:46.735319 2367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047ace075b34786aa7de1ed99571845" candidate_uuid: "210cf01a7431498ba516ad475acb3e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7f425af7cb4538ba206b8145e07d55"
I20260619 11:59:46.735409 2227 raft_consensus.cc:3037] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:46.735441 2367 raft_consensus.cc:3037] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:46.736135 2367 raft_consensus.cc:2454] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210cf01a7431498ba516ad475acb3e90 in term 1.
I20260619 11:59:46.736482 2227 raft_consensus.cc:2454] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 210cf01a7431498ba516ad475acb3e90 in term 1.
I20260619 11:59:46.736727 2018 leader_election.cc:302] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210cf01a7431498ba516ad475acb3e90, fc7f425af7cb4538ba206b8145e07d55; no voters:
I20260619 11:59:46.736898 2444 raft_consensus.cc:2781] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:46.737089 2444 raft_consensus.cc:686] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 1 LEADER]: Becoming Leader. State: Replica: 210cf01a7431498ba516ad475acb3e90, State: Running, Role: LEADER
I20260619 11:59:46.737193 2444 consensus_queue.cc:237] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.738138 1930 catalog_manager.cc:5495] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 210cf01a7431498ba516ad475acb3e90 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "210cf01a7431498ba516ad475acb3e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } health_report { overall_health: UNKNOWN } } }
Jun 19 11:59:46 dist-test-slave-fbq8 krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:46.781508 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.773581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.0.203.254:46117)
0619 11:59:46.773685 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:46.773687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:46.773769 (+ 82us) server_negotiation.cc:407] Connection header received
0619 11:59:46.773847 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:46.773847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:46.773867 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:46.773950 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:46.774581 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.775182 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.775986 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.776177 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.776231 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.778573 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:46.778575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:46.778575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:46.778583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:46.780166 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.780631 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.780632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.780633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.780713 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:46.780925 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:46.780926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:46.780926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:46.781049 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:46.781130 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:46.781285 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 11:59:46.781383 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":67}
I20260619 11:59:46.781584 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:46.773502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46117 (local address 127.0.0.1:36052)
0619 11:59:46.773672 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:46.773691 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:46.773761 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:46.774033 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:46.774034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:46.774385 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:46.774441 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.774443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.775313 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.775314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.775885 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:46.775887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:46.776251 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:46.776252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:46.776296 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:46.776893 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:46.776928 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:46.778454 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:46.780292 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.780302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.780305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.780518 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.780787 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:46.780788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:46.780788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:46.780854 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:46.781120 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:46.781128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:46.781176 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 11:59:46.781261 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:59:46.781449 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":53,"thread_start_us":103,"threads_started":1}
I20260619 11:59:46.784440 1930 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:36052:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:59:46.784729 1930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:46.787024 2347 tablet_service.cc:1467] Processing CreateTablet for tablet e95de19361ea4fdf841e21e23cf80a69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e1b2bf179884ea9a2ca110fcceb6aec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:59:46.787060 2067 tablet_service.cc:1467] Processing CreateTablet for tablet e95de19361ea4fdf841e21e23cf80a69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e1b2bf179884ea9a2ca110fcceb6aec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:59:46.787222 2207 tablet_service.cc:1467] Processing CreateTablet for tablet e95de19361ea4fdf841e21e23cf80a69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e1b2bf179884ea9a2ca110fcceb6aec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:59:46.787266 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95de19361ea4fdf841e21e23cf80a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.787267 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95de19361ea4fdf841e21e23cf80a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.787369 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95de19361ea4fdf841e21e23cf80a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:46.788987 2436 tablet_bootstrap.cc:492] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: Bootstrap starting.
I20260619 11:59:46.788991 2437 tablet_bootstrap.cc:492] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: Bootstrap starting.
I20260619 11:59:46.789012 2438 tablet_bootstrap.cc:492] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: Bootstrap starting.
I20260619 11:59:46.789409 2438 tablet_bootstrap.cc:654] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.789469 2437 tablet_bootstrap.cc:654] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.789525 2436 tablet_bootstrap.cc:654] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:46.790242 2437 tablet_bootstrap.cc:492] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: No bootstrap required, opened a new log
I20260619 11:59:46.790337 2437 ts_tablet_manager.cc:1397] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:46.790341 2438 tablet_bootstrap.cc:492] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: No bootstrap required, opened a new log
I20260619 11:59:46.790395 2438 ts_tablet_manager.cc:1397] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:46.790521 2436 tablet_bootstrap.cc:492] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: No bootstrap required, opened a new log
I20260619 11:59:46.790534 2438 raft_consensus.cc:348] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.790587 2436 ts_tablet_manager.cc:1397] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 11:59:46.790616 2438 raft_consensus.cc:374] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.790546 2437 raft_consensus.cc:348] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.790657 2438 raft_consensus.cc:729] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7f425af7cb4538ba206b8145e07d55, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.790658 2437 raft_consensus.cc:374] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.790697 2437 raft_consensus.cc:729] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210cf01a7431498ba516ad475acb3e90, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.790756 2437 consensus_queue.cc:260] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.790725 2438 consensus_queue.cc:260] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.790776 2436 raft_consensus.cc:348] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.790860 2438 ts_tablet_manager.cc:1428] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 11:59:46.790867 2436 raft_consensus.cc:374] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:46.790863 2437 ts_tablet_manager.cc:1428] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 11:59:46.790903 2436 raft_consensus.cc:729] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59019c95c34d36a5dff384cabd8864, State: Initialized, Role: FOLLOWER
I20260619 11:59:46.790967 2436 consensus_queue.cc:260] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:46.790977 2442 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:46.791042 2442 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:46.791074 2436 ts_tablet_manager.cc:1428] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 11:59:46.791072 2444 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:46.791158 2444 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:46.791224 2441 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:46.791325 2441 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:47.149801 2441 raft_consensus.cc:482] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:47.150182 2441 raft_consensus.cc:504] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:47.150642 2441 leader_election.cc:288] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 210cf01a7431498ba516ad475acb3e90 (127.0.203.193:33883), fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195:39671)
I20260619 11:59:47.155278 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.150880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:33883 (local address 127.0.203.194:41447)
0619 11:59:47.151057 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:47.151070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:47.151151 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:47.151714 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:47.151715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:47.151718 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:47.151798 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.151800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.152522 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.152523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.154055 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.154058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.154786 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.154788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.154988 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.154995 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:59:47.155119 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 11:59:47.155177 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260619 11:59:47.155278 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.151276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:41447 (local address 127.0.203.193:33883)
0619 11:59:47.151554 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.151556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.151564 (+ 8us) server_negotiation.cc:407] Connection header received
0619 11:59:47.151607 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.151607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.151622 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.151695 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:47.151890 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.152427 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.154176 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.154671 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.154788 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.154798 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.155131 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.155198 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":58,"threads_started":1}
I20260619 11:59:47.155937 2087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95de19361ea4fdf841e21e23cf80a69" candidate_uuid: "0d59019c95c34d36a5dff384cabd8864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210cf01a7431498ba516ad475acb3e90" is_pre_election: true
I20260619 11:59:47.156069 2087 raft_consensus.cc:2454] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d59019c95c34d36a5dff384cabd8864 in term 0.
I20260619 11:59:47.156457 2158 leader_election.cc:302] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59019c95c34d36a5dff384cabd8864, 210cf01a7431498ba516ad475acb3e90; no voters:
I20260619 11:59:47.156672 2441 raft_consensus.cc:2781] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:47.156739 2441 raft_consensus.cc:482] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:47.156787 2441 raft_consensus.cc:3037] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:47.156857 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.151119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:39671 (local address 127.0.203.194:35071)
0619 11:59:47.151716 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:47.151733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:47.151869 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:47.152521 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:47.152522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:47.152525 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:47.152661 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.152666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.153856 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.153858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.155327 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.155330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.156184 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.156185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.156271 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.156277 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:59:47.156360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 11:59:47.156411 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":6,"thread_start_us":51,"threads_started":1}
I20260619 11:59:47.156876 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.151454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:35071 (local address 127.0.203.195:39671)
0619 11:59:47.151529 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.151532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.151844 (+ 312us) server_negotiation.cc:407] Connection header received
0619 11:59:47.151983 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.151985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.151998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.152435 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:47.152780 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.153720 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.155483 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.156073 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.156181 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.156193 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.156714 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.156774 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:59:47.157981 2367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95de19361ea4fdf841e21e23cf80a69" candidate_uuid: "0d59019c95c34d36a5dff384cabd8864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7f425af7cb4538ba206b8145e07d55" is_pre_election: true
I20260619 11:59:47.158043 2441 raft_consensus.cc:504] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:47.158106 2367 raft_consensus.cc:2454] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d59019c95c34d36a5dff384cabd8864 in term 0.
I20260619 11:59:47.158169 2441 leader_election.cc:288] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [CANDIDATE]: Term 1 election: Requested vote from peers 210cf01a7431498ba516ad475acb3e90 (127.0.203.193:33883), fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195:39671)
I20260619 11:59:47.158643 2087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95de19361ea4fdf841e21e23cf80a69" candidate_uuid: "0d59019c95c34d36a5dff384cabd8864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210cf01a7431498ba516ad475acb3e90"
I20260619 11:59:47.158756 2087 raft_consensus.cc:3037] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:47.158963 2367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95de19361ea4fdf841e21e23cf80a69" candidate_uuid: "0d59019c95c34d36a5dff384cabd8864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7f425af7cb4538ba206b8145e07d55"
I20260619 11:59:47.159057 2367 raft_consensus.cc:3037] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:47.159967 2087 raft_consensus.cc:2454] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d59019c95c34d36a5dff384cabd8864 in term 1.
I20260619 11:59:47.160024 2367 raft_consensus.cc:2454] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d59019c95c34d36a5dff384cabd8864 in term 1.
I20260619 11:59:47.160234 2158 leader_election.cc:302] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59019c95c34d36a5dff384cabd8864, 210cf01a7431498ba516ad475acb3e90; no voters:
I20260619 11:59:47.160398 2441 raft_consensus.cc:2781] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:47.160530 2441 raft_consensus.cc:686] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [term 1 LEADER]: Becoming Leader. State: Replica: 0d59019c95c34d36a5dff384cabd8864, State: Running, Role: LEADER
I20260619 11:59:47.160625 2441 consensus_queue.cc:237] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } }
I20260619 11:59:47.161125 2480 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: TxnStatusTablet state changed. Reason: New leader 0d59019c95c34d36a5dff384cabd8864. Latest consensus state: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:47.161260 2480 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:59:47.161473 2482 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:59:47.161646 2482 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:59:47.163506 1929 catalog_manager.cc:5495] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194). New cstate: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } health_report { overall_health: UNKNOWN } } }
I20260619 11:59:47.204445 2444 consensus_queue.cc:1035] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:47.207247 2487 consensus_queue.cc:1035] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:47.240509 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.237370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40889 (local address 127.0.0.1:42486)
0619 11:59:47.237463 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:47.237476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:47.237557 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:47.237782 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:47.237783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:47.237784 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:59:47.237848 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.237850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.238746 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.238748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.239428 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.239430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.239878 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.239879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.239949 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.239981 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:59:47.240307 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:59:47.240317 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 11:59:47.240349 (+ 32us) client_negotiation.cc:240] Negotiation successful
0619 11:59:47.240409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:59:47.240513 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.237445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.0.203.194:40889)
0619 11:59:47.237561 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.237563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.237573 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:59:47.237652 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.237653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.237692 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.237779 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:47.237935 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.238597 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.239539 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.239781 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.239878 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.240104 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:47.240235 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:47.240330 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.240375 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.240402 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":48,"threads_started":1}
I20260619 11:59:47.242069 2367 raft_consensus.cc:1261] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Refusing update from remote peer 0d59019c95c34d36a5dff384cabd8864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:47.242077 2087 raft_consensus.cc:1261] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Refusing update from remote peer 0d59019c95c34d36a5dff384cabd8864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:47.242468 2480 consensus_queue.cc:1035] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:47.242655 2441 consensus_queue.cc:1035] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:47.243479 2442 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: TxnStatusTablet state changed. Reason: New leader 0d59019c95c34d36a5dff384cabd8864. Latest consensus state: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:47.243579 2442 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:47.243842 2487 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: TxnStatusTablet state changed. Reason: New leader 0d59019c95c34d36a5dff384cabd8864. Latest consensus state: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:47.243953 2487 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:47.244664 2486 mvcc.cc:205] Tried to move back new op lower bound from 7298541106141048832 to 7298541105811038208. Current Snapshot: MvccSnapshot[applied={T|T < 7298541106141048832}]
I20260619 11:59:47.244766 2442 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:47.244822 2442 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P fc7f425af7cb4538ba206b8145e07d55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:47.245103 2489 mvcc.cc:205] Tried to move back new op lower bound from 7298541106141048832 to 7298541105811038208. Current Snapshot: MvccSnapshot[applied={T|T < 7298541106141048832 or (T in {7298541106141048832})}]
I20260619 11:59:47.245638 2480 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:47.245725 2480 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:59:47.246201 2487 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:47.246299 2487 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P 210cf01a7431498ba516ad475acb3e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:47.244598 2441 tablet_replica.cc:439] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d59019c95c34d36a5dff384cabd8864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210cf01a7431498ba516ad475acb3e90" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 33883 } } peers { permanent_uuid: "0d59019c95c34d36a5dff384cabd8864" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40889 } } peers { permanent_uuid: "fc7f425af7cb4538ba206b8145e07d55" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 39671 } } }
I20260619 11:59:47.246507 2441 tablet_replica.cc:442] T e95de19361ea4fdf841e21e23cf80a69 P 0d59019c95c34d36a5dff384cabd8864: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:59:47.249940 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.247267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.0.203.193:33883)
0619 11:59:47.247319 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.247321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.247348 (+ 27us) server_negotiation.cc:407] Connection header received
0619 11:59:47.247447 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.247447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.247470 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.247555 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:47.247719 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.248234 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.249105 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.249319 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.249408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.249584 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:47.249704 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:47.249792 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.249834 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.249856 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 11:59:47.255997 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.253012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:55689 (local address 127.0.203.194:40889)
0619 11:59:47.253061 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.253063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.253153 (+ 90us) server_negotiation.cc:407] Connection header received
0619 11:59:47.253246 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.253247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.253259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.253320 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:47.253581 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.254051 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.255244 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.255432 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.255505 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.255729 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:47.255760 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:47.255817 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.255919 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.255941 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 11:59:47.256054 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.252895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40889 (local address 127.0.203.193:55689)
0619 11:59:47.253086 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:47.253098 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:47.253170 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:47.253361 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:47.253362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:47.253364 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:59:47.253428 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.253430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.254156 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.254157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.255136 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.255139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.255530 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.255531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.255597 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.255613 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:59:47.255846 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:59:47.255847 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 11:59:47.255915 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:59:47.255959 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260619 11:59:47.268005 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.265007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:49161 (local address 127.0.203.193:33883)
0619 11:59:47.265054 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.265057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.265127 (+ 70us) server_negotiation.cc:407] Connection header received
0619 11:59:47.265198 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.265199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.265209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.265269 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:47.265484 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.265985 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.267190 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.267375 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.267431 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.267685 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:47.267712 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:47.267790 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.267913 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.267938 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 11:59:47.268054 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.264812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:33883 (local address 127.0.203.194:49161)
0619 11:59:47.265055 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:47.265070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:47.265139 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:47.265273 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:47.265274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:47.265276 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:59:47.265366 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.265368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.266095 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.266096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.267091 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:47.267094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.267474 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.267475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:47.267544 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.267558 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:59:47.267826 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:59:47.267828 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 11:59:47.267893 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:59:47.267939 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260619 11:59:47.278563 1928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:47.278714 1928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:47.279433 1928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91efef3053c64f48ad5c7f18facc10e5: Sending DeleteTablet for 3 replicas of tablet c047ace075b34786aa7de1ed99571845
I20260619 11:59:47.279768 2207 tablet_service.cc:1514] Processing DeleteTablet for tablet c047ace075b34786aa7de1ed99571845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:47 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:42472
I20260619 11:59:47.279819 2347 tablet_service.cc:1514] Processing DeleteTablet for tablet c047ace075b34786aa7de1ed99571845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:47 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:40854
I20260619 11:59:47.280180 2508 tablet_replica.cc:330] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: stopping tablet replica
I20260619 11:59:47.280172 2067 tablet_service.cc:1514] Processing DeleteTablet for tablet c047ace075b34786aa7de1ed99571845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:47 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44192
I20260619 11:59:47.280304 2508 raft_consensus.cc:2229] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:47.280390 2508 raft_consensus.cc:2258] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:47.280807 2508 ts_tablet_manager.cc:1905] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:47.281019 2509 tablet_replica.cc:330] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: stopping tablet replica
I20260619 11:59:47.281193 2509 raft_consensus.cc:2229] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:59:47.281389 2509 raft_consensus.cc:2258] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:47.281949 2507 tablet_replica.cc:330] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: stopping tablet replica
I20260619 11:59:47.282029 2507 raft_consensus.cc:2229] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:47.282080 2507 raft_consensus.cc:2258] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:47.282698 2509 ts_tablet_manager.cc:1905] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:47.283013 2507 ts_tablet_manager.cc:1905] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:47.285519 2509 ts_tablet_manager.cc:1918] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:47.285622 2509 log.cc:1196] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/c047ace075b34786aa7de1ed99571845
I20260619 11:59:47.285969 2509 ts_tablet_manager.cc:1939] T c047ace075b34786aa7de1ed99571845 P 210cf01a7431498ba516ad475acb3e90: Deleting consensus metadata
I20260619 11:59:47.286610 1914 catalog_manager.cc:4841] TS 210cf01a7431498ba516ad475acb3e90 (127.0.203.193:33883): tablet c047ace075b34786aa7de1ed99571845 (table test-table [id=a58accc8e48f49f58b15e3e226709828]) successfully deleted
I20260619 11:59:47.287400 2507 ts_tablet_manager.cc:1918] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:47.287504 2507 log.cc:1196] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/c047ace075b34786aa7de1ed99571845
I20260619 11:59:47.287781 2507 ts_tablet_manager.cc:1939] T c047ace075b34786aa7de1ed99571845 P 0d59019c95c34d36a5dff384cabd8864: Deleting consensus metadata
I20260619 11:59:47.288506 2508 ts_tablet_manager.cc:1918] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:47.288508 1916 catalog_manager.cc:4841] TS 0d59019c95c34d36a5dff384cabd8864 (127.0.203.194:40889): tablet c047ace075b34786aa7de1ed99571845 (table test-table [id=a58accc8e48f49f58b15e3e226709828]) successfully deleted
I20260619 11:59:47.288582 2508 log.cc:1196] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/c047ace075b34786aa7de1ed99571845
I20260619 11:59:47.288826 2508 ts_tablet_manager.cc:1939] T c047ace075b34786aa7de1ed99571845 P fc7f425af7cb4538ba206b8145e07d55: Deleting consensus metadata
I20260619 11:59:47.289924 1914 catalog_manager.cc:4841] TS fc7f425af7cb4538ba206b8145e07d55 (127.0.203.195:39671): tablet c047ace075b34786aa7de1ed99571845 (table test-table [id=a58accc8e48f49f58b15e3e226709828]) successfully deleted
Jun 19 11:59:47 dist-test-slave-fbq8 krb5kdc[1885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 11:59:47.299383 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.281761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.0.203.193:33883)
0619 11:59:47.281813 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.281815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.288908 (+ 7093us) server_negotiation.cc:407] Connection header received
0619 11:59:47.288958 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.288959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.288975 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.289045 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:47.289696 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.290412 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.291335 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.291630 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.291672 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.295356 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:47.295364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:47.295365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:47.295372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:47.297761 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:47.298225 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:47.298229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:47.298230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:47.298326 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:47.298579 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:47.298580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:47.298580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:47.298769 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:47.298904 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.299160 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.299259 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 11:59:47.300791 2127 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44206
I20260619 11:59:47.311666 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:47.303435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.0.203.254:46117)
0619 11:59:47.303668 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:47.303670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:47.303682 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:59:47.303760 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:47.303760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:47.303775 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:47.303845 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:47.304489 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.305262 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.306050 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:47.306295 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:47.306394 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:47.307777 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:47.307778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:47.307779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:47.307787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:47.309568 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:47.309978 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:47.309979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:47.309980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:47.310046 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:47.310272 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:47.310274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:47.310274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:47.310408 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:47.310538 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:47.310651 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 11:59:47.310744 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":59,"threads_started":1}
W20260619 11:59:47.311949 1928 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36068
I20260619 11:59:47.312760 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 2002
I20260619 11:59:47.320449 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 2142
I20260619 11:59:47.328470 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 2282
I20260619 11:59:47.334985 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 1898
2026-06-19T11:59:47Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2913 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:47 dist-test-slave-fbq8 krb5kdc[2517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:47 dist-test-slave-fbq8 krb5kdc[2517](info): set up 2 sockets
Jun 19 11:59:47 dist-test-slave-fbq8 krb5kdc[2517](info): commencing operation
krb5kdc: starting...
W20260619 11:59:48.416528 815 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:48 dist-test-slave-fbq8 krb5kdc[2517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:48.558583 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36401
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:36401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:48.649464 2530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:48.649691 2530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:48.649722 2530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:48.650987 2530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:48.651039 2530 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:59:48.651053 2530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:48.651067 2530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:48.651079 2530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:48.651093 2530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:48.652730 2530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:36401
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:48.653014 2530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:48.653290 2530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:48.659075 2536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:48.659175 2538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:48.659075 2535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:48.659428 2530 server_base.cc:1034] running on GCE node
I20260619 11:59:48.659727 2530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:48.660109 2530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:48.661288 2530 hybrid_clock.cc:648] HybridClock initialized: now 1781870388661258 us; error 50 us; skew 500 ppm
Jun 19 11:59:48 dist-test-slave-fbq8 krb5kdc[2517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:48.664345 2530 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:48.665028 2530 webserver.cc:460] Webserver started at http://127.0.203.254:43351/ using document root <none> and password file <none>
I20260619 11:59:48.665302 2530 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:48.665362 2530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:48.665472 2530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:48.666774 2530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "cccefb01dbf6463ba6d431ec119597ad"
format_stamp: "Formatted at 2026-06-19 11:59:48 on dist-test-slave-fbq8"
server_key: "01d3b8e477f98240a85e9ae6e7288e6f"
server_key_iv: "683dba4a7bb8e35862148cb4e4e18c14"
server_key_version: "encryptionkey@0"
I20260619 11:59:48.667212 2530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "cccefb01dbf6463ba6d431ec119597ad"
format_stamp: "Formatted at 2026-06-19 11:59:48 on dist-test-slave-fbq8"
server_key: "01d3b8e477f98240a85e9ae6e7288e6f"
server_key_iv: "683dba4a7bb8e35862148cb4e4e18c14"
server_key_version: "encryptionkey@0"
I20260619 11:59:48.668992 2530 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:59:48.670203 2545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:48.670557 2530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:48.670714 2530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "cccefb01dbf6463ba6d431ec119597ad"
format_stamp: "Formatted at 2026-06-19 11:59:48 on dist-test-slave-fbq8"
server_key: "01d3b8e477f98240a85e9ae6e7288e6f"
server_key_iv: "683dba4a7bb8e35862148cb4e4e18c14"
server_key_version: "encryptionkey@0"
I20260619 11:59:48.670861 2530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:48.680836 2530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:48.683336 2530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:48.683516 2530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:48.689244 2530 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:36401
I20260619 11:59:48.689337 2607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:36401 every 8 connection(s)
I20260619 11:59:48.689716 2530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:48.690541 2608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:48.693393 2608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Bootstrap starting.
I20260619 11:59:48.694132 2608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:48.694484 2608 log.cc:824] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:48.695112 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 2530
I20260619 11:59:48.695353 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:48.695554 815 external_mini_cluster.cc:1428] Setting key 2bf992ce5dd3a86a8274b0cccd02a445
I20260619 11:59:48.695708 2608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: No bootstrap required, opened a new log
I20260619 11:59:48.698691 2608 raft_consensus.cc:348] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccefb01dbf6463ba6d431ec119597ad" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36401 } }
I20260619 11:59:48.698861 2608 raft_consensus.cc:374] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:48.698885 2608 raft_consensus.cc:729] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cccefb01dbf6463ba6d431ec119597ad, State: Initialized, Role: FOLLOWER
I20260619 11:59:48.699052 2608 consensus_queue.cc:260] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccefb01dbf6463ba6d431ec119597ad" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36401 } }
I20260619 11:59:48.699141 2608 raft_consensus.cc:388] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:48.699173 2608 raft_consensus.cc:482] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:48.699234 2608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:48.699919 2608 raft_consensus.cc:504] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccefb01dbf6463ba6d431ec119597ad" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36401 } }
I20260619 11:59:48.700073 2608 leader_election.cc:302] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cccefb01dbf6463ba6d431ec119597ad; no voters:
I20260619 11:59:48.700296 2608 leader_election.cc:288] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:48.700348 2613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:48.700489 2613 raft_consensus.cc:686] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [term 1 LEADER]: Becoming Leader. State: Replica: cccefb01dbf6463ba6d431ec119597ad, State: Running, Role: LEADER
I20260619 11:59:48.700767 2608 sys_catalog.cc:564] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:48.700733 2613 consensus_queue.cc:237] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccefb01dbf6463ba6d431ec119597ad" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36401 } }
I20260619 11:59:48.701368 2614 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cccefb01dbf6463ba6d431ec119597ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccefb01dbf6463ba6d431ec119597ad" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36401 } } }
I20260619 11:59:48.701640 2614 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:48.702033 2622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:48.702458 2615 sys_catalog.cc:455] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader cccefb01dbf6463ba6d431ec119597ad. Latest consensus state: current_term: 1 leader_uuid: "cccefb01dbf6463ba6d431ec119597ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cccefb01dbf6463ba6d431ec119597ad" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36401 } } }
I20260619 11:59:48.702811 2615 sys_catalog.cc:458] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:48.703328 2622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:59:48.705632 2622 catalog_manager.cc:1347] Generated new cluster ID: 677f753ab6a94a718c2437006f4886df
I20260619 11:59:48.705705 2622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 11:59:48 dist-test-slave-fbq8 krb5kdc[2517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:48.711130 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:48.696235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.0.203.254:36401)
0619 11:59:48.696508 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:48.696518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:48.696551 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:59:48.698753 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:48.698765 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:48.698849 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:48.699126 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:48.700103 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.702031 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.703252 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.703977 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.704150 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:48.707439 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:48.707445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:48.707445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:48.707465 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:48.709393 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:48.709800 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:48.709804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:48.709805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:48.709929 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:48.710168 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:48.710169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:48.710170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:48.710448 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:48.710647 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:48.710734 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 11:59:48.710904 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":147,"threads_started":1}
I20260619 11:59:48.720810 2622 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:48.721552 2622 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:48.725096 2622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Generated new TSK 0
I20260619 11:59:48.725324 2622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 11:59:48.766290 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36401
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:48.880407 2634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:48.880735 2634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:48.880790 2634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:48.882131 2634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:48.882227 2634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:48.882272 2634 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:59:48.882308 2634 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:59:48.882350 2634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 11:59:48.884018 2634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:36401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:48.884353 2634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:48.884706 2634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:48.891654 2640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:48.891650 2639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:48.891754 2642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:48.892400 2634 server_base.cc:1034] running on GCE node
I20260619 11:59:48.892829 2634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:48.893105 2634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:48.894305 2634 hybrid_clock.cc:648] HybridClock initialized: now 1781870388894266 us; error 78 us; skew 500 ppm
Jun 19 11:59:48 dist-test-slave-fbq8 krb5kdc[2517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:48.897486 2634 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 11:59:48.898135 2634 webserver.cc:460] Webserver started at http://127.0.203.193:36587/ using document root <none> and password file <none>
I20260619 11:59:48.898439 2634 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:48.899201 2634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:48.899407 2634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:48.900784 2634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "d2643e7f8e474732a6198382b99c02dc"
format_stamp: "Formatted at 2026-06-19 11:59:48 on dist-test-slave-fbq8"
server_key: "137a6a623a6f8944eb1f230bd13045bc"
server_key_iv: "efac28c01eafb4b653c54be188e23724"
server_key_version: "encryptionkey@0"
I20260619 11:59:48.901889 2634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "d2643e7f8e474732a6198382b99c02dc"
format_stamp: "Formatted at 2026-06-19 11:59:48 on dist-test-slave-fbq8"
server_key: "137a6a623a6f8944eb1f230bd13045bc"
server_key_iv: "efac28c01eafb4b653c54be188e23724"
server_key_version: "encryptionkey@0"
I20260619 11:59:48.904273 2634 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:59:48.905772 2649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:48.906196 2634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:48.906364 2634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "d2643e7f8e474732a6198382b99c02dc"
format_stamp: "Formatted at 2026-06-19 11:59:48 on dist-test-slave-fbq8"
server_key: "137a6a623a6f8944eb1f230bd13045bc"
server_key_iv: "efac28c01eafb4b653c54be188e23724"
server_key_version: "encryptionkey@0"
I20260619 11:59:48.906490 2634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:48.922433 2634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:48.926805 2634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:48.927034 2634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:48.927727 2634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:48.927798 2634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:48.927841 2634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:48.927870 2634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:59:48 dist-test-slave-fbq8 krb5kdc[2517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:48.938959 2634 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:40549
I20260619 11:59:48.939365 2764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:40549 every 8 connection(s)
I20260619 11:59:48.939864 2634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 11:59:48.943198 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:48.929135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:35895 (local address 127.0.203.254:36401)
0619 11:59:48.929248 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:48.929251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:48.929480 (+ 229us) server_negotiation.cc:407] Connection header received
0619 11:59:48.930462 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:48.930464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:48.930484 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:48.930543 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:48.931833 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.932667 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.934389 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.934750 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.934851 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:48.938442 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:48.938445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:48.938445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:48.938453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:48.941212 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:48.941773 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:48.941775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:48.941776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:48.941859 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:48.942284 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:48.942285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:48.942285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:48.942434 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:48.942512 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:48.943020 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 11:59:48.943126 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 11:59:48.943401 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:48.929014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36401 (local address 127.0.203.193:35895)
0619 11:59:48.929305 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:48.929355 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:48.930300 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:48.930626 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:48.930633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:48.931320 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:48.931658 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:48.931664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.932798 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.932799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:48.934296 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:48.934300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.934917 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.934919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:48.935082 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:48.935895 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:48.935905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:48.938315 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:48.941355 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:48.941361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:48.941363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:48.941635 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:48.942019 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:48.942020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:48.942020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:48.942164 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:48.942547 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:48.942554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:48.942699 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 11:59:48.942862 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 11:59:48.943216 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":123,"threads_started":1}
I20260619 11:59:48.944401 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 2634
I20260619 11:59:48.944418 2765 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36401
I20260619 11:59:48.944494 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 11:59:48.944504 2765 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:48.944716 815 external_mini_cluster.cc:1428] Setting key 395040481045a36ec1350921fb1a6f96
I20260619 11:59:48.944768 2765 heartbeater.cc:511] Master 127.0.203.254:36401 requested a full tablet report, sending...
I20260619 11:59:48.946174 2562 ts_manager.cc:194] Registered new tserver with Master: d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549)
I20260619 11:59:48.947225 2562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:35895
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 11:59:48.955097 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:48.945748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:60565 (local address 127.0.203.254:36401)
0619 11:59:48.947507 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:48.947510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:48.947534 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:59:48.947567 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:48.947568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:48.947584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:48.947689 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:48.948301 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.948934 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.949769 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.949987 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.950090 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:48.951278 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:48.951280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:48.951280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:48.951288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:48.953500 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:48.953970 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:48.953971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:48.953971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:48.954057 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:48.954436 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:48.954437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:48.954437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:48.954626 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:48.954701 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:48.954879 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 11:59:48.955003 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727}
I20260619 11:59:48.955101 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:48.945758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36401 (local address 127.0.203.193:60565)
0619 11:59:48.945946 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:48.945958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:48.946058 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:48.947749 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:48.947751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:48.948070 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:48.948152 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:48.948155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.949051 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.949052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:48.949647 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:48.949650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:48.950089 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:48.950090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:48.950172 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:48.950765 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:48.950770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:48.951140 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:48.953657 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:48.953659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:48.953659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:48.953875 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:48.954261 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:48.954262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:48.954263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:48.954344 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:48.954705 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:48.954706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:48.954761 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 11:59:48.954853 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 11:59:48.954991 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 11:59:48.982263 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36401
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:49.097025 2774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:49.097338 2774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:49.097395 2774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:49.098905 2774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:49.098973 2774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:49.098999 2774 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:59:49.099013 2774 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:59:49.099035 2774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 11:59:49.101213 2774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:36401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:49.101464 2774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:49.101691 2774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:49.107939 2779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:49.108103 2774 server_base.cc:1034] running on GCE node
W20260619 11:59:49.107931 2780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:49.107931 2782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:49.108568 2774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:49.108976 2774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:49.110191 2774 hybrid_clock.cc:648] HybridClock initialized: now 1781870389110128 us; error 80 us; skew 500 ppm
Jun 19 11:59:49 dist-test-slave-fbq8 krb5kdc[2517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:49.113811 2774 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 11:59:49.114425 2774 webserver.cc:460] Webserver started at http://127.0.203.194:34115/ using document root <none> and password file <none>
I20260619 11:59:49.114699 2774 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:49.114804 2774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:49.114974 2774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:49.116339 2774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "628cfe85868b49ab8616017ae22431bc"
format_stamp: "Formatted at 2026-06-19 11:59:49 on dist-test-slave-fbq8"
server_key: "03e5698e9af28c359bc027147502e4e5"
server_key_iv: "9f00b0b220de51c7b8cac040c2e78a8c"
server_key_version: "encryptionkey@0"
I20260619 11:59:49.116843 2774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "628cfe85868b49ab8616017ae22431bc"
format_stamp: "Formatted at 2026-06-19 11:59:49 on dist-test-slave-fbq8"
server_key: "03e5698e9af28c359bc027147502e4e5"
server_key_iv: "9f00b0b220de51c7b8cac040c2e78a8c"
server_key_version: "encryptionkey@0"
I20260619 11:59:49.118506 2774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:49.119594 2789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:49.119952 2774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:49.120096 2774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "628cfe85868b49ab8616017ae22431bc"
format_stamp: "Formatted at 2026-06-19 11:59:49 on dist-test-slave-fbq8"
server_key: "03e5698e9af28c359bc027147502e4e5"
server_key_iv: "9f00b0b220de51c7b8cac040c2e78a8c"
server_key_version: "encryptionkey@0"
I20260619 11:59:49.120199 2774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:49.131758 2774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:49.134368 2774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:49.134553 2774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:49.135246 2774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:49.135301 2774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:49.135336 2774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:49.135362 2774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:59:49 dist-test-slave-fbq8 krb5kdc[2517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:49.143496 2774 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:40461
I20260619 11:59:49.143536 2904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:40461 every 8 connection(s)
I20260619 11:59:49.143971 2774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 11:59:49.146351 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.135802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:47911 (local address 127.0.203.254:36401)
0619 11:59:49.135884 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.135887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.136101 (+ 214us) server_negotiation.cc:407] Connection header received
0619 11:59:49.136629 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.136631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.136663 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.136740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:49.137817 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.138664 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.139921 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.140139 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.140232 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.142873 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:49.142875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:49.142875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:49.142884 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:49.144852 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.145309 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.145310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.145310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.145373 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.145637 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.145638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.145638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.145769 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:49.145852 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.146148 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.146241 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:59:49.146566 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.135760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36401 (local address 127.0.203.194:47911)
0619 11:59:49.135955 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.135996 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.136526 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.137037 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.137044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.137395 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:49.137695 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.137700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.138821 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.138822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.139756 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.139759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.140252 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.140254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.140414 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.141135 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:49.141145 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:49.142761 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:49.144955 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.144961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.144963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.145206 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.145426 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.145427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.145427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.145545 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.145840 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:49.145846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:49.145999 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.146127 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.146310 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260619 11:59:49.147148 2905 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36401
I20260619 11:59:49.147245 2905 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:49.147459 2905 heartbeater.cc:511] Master 127.0.203.254:36401 requested a full tablet report, sending...
I20260619 11:59:49.148072 2561 ts_manager.cc:194] Registered new tserver with Master: 628cfe85868b49ab8616017ae22431bc (127.0.203.194:40461)
I20260619 11:59:49.148362 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 2774
I20260619 11:59:49.148473 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 11:59:49.148587 2561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:47911
I20260619 11:59:49.149111 815 external_mini_cluster.cc:1428] Setting key 29cf43a4b0d8a61fb1ea0d3e5f28cecf
I20260619 11:59:49.155898 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.148944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:53935 (local address 127.0.203.254:36401)
0619 11:59:49.149027 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.149029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.149157 (+ 128us) server_negotiation.cc:407] Connection header received
0619 11:59:49.149234 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.149235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.149247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.149304 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:49.149810 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.150350 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.151514 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.151726 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.151817 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.153054 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:49.153056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:49.153056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:49.153064 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:49.154476 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.154971 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.154972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.154972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.155038 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.155328 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.155329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.155329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.155485 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:49.155549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.155766 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.155832 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:59:49.156008 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.148849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36401 (local address 127.0.203.194:53935)
0619 11:59:49.149060 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.149073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.149172 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.149311 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.149312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.149587 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:49.149657 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.149660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.150464 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.150465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.151380 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.151383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.151818 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.151819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.151886 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.152456 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:49.152461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:49.152954 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:49.154576 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.154577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.154578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.154856 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.155133 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.155134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.155134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.155213 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.155572 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:49.155573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:49.155638 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.155734 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.155869 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 11:59:49.181092 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36401
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:49.266953 2914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:49.267197 2914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:49.267225 2914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:49.268430 2914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:49.268489 2914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:49.268512 2914 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:59:49.268524 2914 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:59:49.268543 2914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 11:59:49.270115 2914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:36401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:49.270336 2914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:49.270593 2914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:49.276393 2919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:49.276453 2922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:49.276453 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:49.276616 2914 server_base.cc:1034] running on GCE node
I20260619 11:59:49.276938 2914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:49.277243 2914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:49.278447 2914 hybrid_clock.cc:648] HybridClock initialized: now 1781870389278414 us; error 59 us; skew 500 ppm
Jun 19 11:59:49 dist-test-slave-fbq8 krb5kdc[2517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:49.281498 2914 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 11:59:49.282042 2914 webserver.cc:460] Webserver started at http://127.0.203.195:36637/ using document root <none> and password file <none>
I20260619 11:59:49.282286 2914 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:49.282358 2914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:49.282533 2914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:49.283696 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "ded06fb1ef014f40ba802be657332422"
format_stamp: "Formatted at 2026-06-19 11:59:49 on dist-test-slave-fbq8"
server_key: "4ab5c6fa7aac786c37fc5580118f0362"
server_key_iv: "3b32c3c1d468fc3ad6ae29afe597bfd5"
server_key_version: "encryptionkey@0"
I20260619 11:59:49.284081 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "ded06fb1ef014f40ba802be657332422"
format_stamp: "Formatted at 2026-06-19 11:59:49 on dist-test-slave-fbq8"
server_key: "4ab5c6fa7aac786c37fc5580118f0362"
server_key_iv: "3b32c3c1d468fc3ad6ae29afe597bfd5"
server_key_version: "encryptionkey@0"
I20260619 11:59:49.285720 2914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 11:59:49.286680 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:49.286960 2914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:49.287117 2914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "ded06fb1ef014f40ba802be657332422"
format_stamp: "Formatted at 2026-06-19 11:59:49 on dist-test-slave-fbq8"
server_key: "4ab5c6fa7aac786c37fc5580118f0362"
server_key_iv: "3b32c3c1d468fc3ad6ae29afe597bfd5"
server_key_version: "encryptionkey@0"
I20260619 11:59:49.287245 2914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:49.327348 2914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:49.330127 2914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:49.330338 2914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:49.330904 2914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:49.330969 2914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:49.331007 2914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:49.331038 2914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:59:49 dist-test-slave-fbq8 krb5kdc[2517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:49.339198 2914 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:36873
I20260619 11:59:49.339303 3044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:36873 every 8 connection(s)
I20260619 11:59:49.339680 2914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 11:59:49.343093 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.331682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:54037 (local address 127.0.203.254:36401)
0619 11:59:49.331775 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.331778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.332043 (+ 265us) server_negotiation.cc:407] Connection header received
0619 11:59:49.332874 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.332875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.332890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.332962 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:49.334081 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.334687 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.335790 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.335993 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.336086 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.339113 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:49.339115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:49.339115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:49.339121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:49.341233 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.342016 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.342017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.342017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.342079 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.342395 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.342396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.342396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.342525 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:49.342591 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.342943 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.343016 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:59:49.343366 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.331645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36401 (local address 127.0.203.195:54037)
0619 11:59:49.331881 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.331930 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.332695 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.333038 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.333043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.333498 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:49.333943 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.333950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.334789 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.334790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.335675 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.335677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.336109 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.336110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.336266 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.337030 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:49.337041 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:49.338993 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:49.341351 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.341356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.341358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.341874 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.342173 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.342174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.342174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.342315 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.342605 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:49.342610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:49.342751 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.342902 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.343059 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":154,"threads_started":1}
I20260619 11:59:49.344035 3045 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36401
I20260619 11:59:49.344137 3045 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:49.344350 3045 heartbeater.cc:511] Master 127.0.203.254:36401 requested a full tablet report, sending...
I20260619 11:59:49.344839 2562 ts_manager.cc:194] Registered new tserver with Master: ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873)
I20260619 11:59:49.345283 2562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:54037
I20260619 11:59:49.348114 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 2914
I20260619 11:59:49.348240 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 11:59:49.348433 815 external_mini_cluster.cc:1428] Setting key 609fecd0508652461dd67faa3ba52948
I20260619 11:59:49.349478 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:59:49.352169 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.345325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36401 (local address 127.0.203.195:48601)
0619 11:59:49.345600 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.345613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.345706 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.345827 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.345828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.346040 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:49.346118 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.346120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.346808 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.346809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.347472 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.347474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.347853 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.347854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.347926 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.348524 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:49.348531 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:49.349112 (+ 581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:49.350877 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.350878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.350879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.351145 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.351412 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.351413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.351413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.351488 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.351871 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:49.351871 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:49.351925 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.351985 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.352088 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
I20260619 11:59:49.352279 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.345428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:48601 (local address 127.0.203.254:36401)
0619 11:59:49.345489 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.345491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.345691 (+ 200us) server_negotiation.cc:407] Connection header received
0619 11:59:49.345764 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.345764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.345775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.345824 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:49.346208 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.346701 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.347562 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.347763 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.347850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.349197 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:49.349198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:49.349199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:49.349205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:49.350772 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.351251 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.351253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.351253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.351320 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.351601 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.351602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.351603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.351785 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:49.351872 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.352047 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.352148 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 11:59:49 dist-test-slave-fbq8 krb5kdc[2517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:59:49 dist-test-slave-fbq8 krb5kdc[2517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:49.380051 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.369587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.0.203.254:36401)
0619 11:59:49.369661 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.369663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.369676 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:59:49.369758 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.369759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.369770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.369875 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:49.370479 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.371018 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.371925 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.372109 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.372181 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.375426 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:49.375428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:49.375428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:49.375435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:49.378633 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.379140 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.379141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.379142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.379200 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.379489 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.379490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.379490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.379617 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:49.379712 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.379903 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.379977 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:59:49.382575 2562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:59:49.384482 2562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:49.393347 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.388347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40461 (local address 127.0.0.1:52480)
0619 11:59:49.388498 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.388511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.388568 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.389087 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.389088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.389091 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:49.389143 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.389144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.390222 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.390224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.391896 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.391899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.392947 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.393024 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.393030 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.393084 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.393190 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":192,"thread_start_us":78,"threads_started":1}
I20260619 11:59:49.393357 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.388153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48716 (local address 127.0.203.195:36873)
0619 11:59:49.388575 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.388581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.388593 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:59:49.388665 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.388668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.388686 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.388777 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:49.388996 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.390212 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392008 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.392776 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392847 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.392909 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.393176 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.393240 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":100,"threads_started":1}
I20260619 11:59:49.393352 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.388147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40549 (local address 127.0.0.1:38974)
0619 11:59:49.388390 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.388402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.388460 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.388967 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.388968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.388970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:49.389060 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.389062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.390414 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.390415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.391829 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.391832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.392859 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.392966 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.392984 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.393115 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.393194 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":111,"threads_started":1}
I20260619 11:59:49.393743 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.388572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.0.203.194:40461)
0619 11:59:49.388907 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.388911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.388922 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:59:49.388958 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.388961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.388975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.389074 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:49.389219 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.390138 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392005 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.392785 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392847 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.392909 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.393584 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.393641 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":109,"threads_started":1}
I20260619 11:59:49.393884 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.387909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36873 (local address 127.0.0.1:48716)
0619 11:59:49.388078 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.388126 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.388259 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.388812 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.388815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.388818 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:49.388895 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.388897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.390309 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.390310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.391898 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.391899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.392948 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.393029 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.393032 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.393096 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.393734 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":810,"thread_start_us":60,"threads_started":1}
I20260619 11:59:49.393896 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.388302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.0.203.193:40549)
0619 11:59:49.388770 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.388775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.388787 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:59:49.388834 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.388836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.388854 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.389134 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:49.389191 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.390282 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.391978 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.392594 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.392777 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.392935 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.393678 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.393790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":167,"threads_started":1}
I20260619 11:59:49.394563 2979 tablet_service.cc:1467] Processing CreateTablet for tablet 68aabdc777a54959becc07266aad93c8 (DEFAULT_TABLE table=test-table [id=f23b867baa0d4fc29115ff62cd10841b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:49.394572 2699 tablet_service.cc:1467] Processing CreateTablet for tablet 68aabdc777a54959becc07266aad93c8 (DEFAULT_TABLE table=test-table [id=f23b867baa0d4fc29115ff62cd10841b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:49.394573 2832 tablet_service.cc:1467] Processing CreateTablet for tablet 68aabdc777a54959becc07266aad93c8 (DEFAULT_TABLE table=test-table [id=f23b867baa0d4fc29115ff62cd10841b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:49.394986 2979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68aabdc777a54959becc07266aad93c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.395154 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68aabdc777a54959becc07266aad93c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.395154 2699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68aabdc777a54959becc07266aad93c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.397837 3068 tablet_bootstrap.cc:492] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Bootstrap starting.
I20260619 11:59:49.398772 3068 tablet_bootstrap.cc:654] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.399107 3068 log.cc:824] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:49.399899 3068 tablet_bootstrap.cc:492] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: No bootstrap required, opened a new log
I20260619 11:59:49.399986 3068 ts_tablet_manager.cc:1397] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:59:49.400753 3071 tablet_bootstrap.cc:492] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Bootstrap starting.
I20260619 11:59:49.400944 3070 tablet_bootstrap.cc:492] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Bootstrap starting.
I20260619 11:59:49.401517 3071 tablet_bootstrap.cc:654] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.401873 3070 tablet_bootstrap.cc:654] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.401975 3071 log.cc:824] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:49.402084 3068 raft_consensus.cc:348] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.402262 3070 log.cc:824] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:49.402283 3068 raft_consensus.cc:374] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.402328 3068 raft_consensus.cc:729] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded06fb1ef014f40ba802be657332422, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.402468 3068 consensus_queue.cc:260] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.402753 3068 ts_tablet_manager.cc:1428] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 11:59:49.402900 3045 heartbeater.cc:503] Master 127.0.203.254:36401 was elected leader, sending a full tablet report...
I20260619 11:59:49.403244 3071 tablet_bootstrap.cc:492] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: No bootstrap required, opened a new log
I20260619 11:59:49.403357 3071 ts_tablet_manager.cc:1397] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 11:59:49.405110 3071 raft_consensus.cc:348] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.405293 3071 raft_consensus.cc:374] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.405342 3071 raft_consensus.cc:729] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.405508 3071 consensus_queue.cc:260] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.405817 3071 ts_tablet_manager.cc:1428] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 11:59:49.405916 2905 heartbeater.cc:503] Master 127.0.203.254:36401 was elected leader, sending a full tablet report...
I20260619 11:59:49.406703 3070 tablet_bootstrap.cc:492] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: No bootstrap required, opened a new log
I20260619 11:59:49.406797 3070 ts_tablet_manager.cc:1397] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 11:59:49.408711 3070 raft_consensus.cc:348] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.408895 3070 raft_consensus.cc:374] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.408921 3070 raft_consensus.cc:729] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2643e7f8e474732a6198382b99c02dc, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.409061 3070 consensus_queue.cc:260] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.409405 3070 ts_tablet_manager.cc:1428] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 11:59:49.409412 2765 heartbeater.cc:503] Master 127.0.203.254:36401 was elected leader, sending a full tablet report...
W20260619 11:59:49.441244 2766 tablet.cc:2369] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:59:49.487352 3075 raft_consensus.cc:482] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:49.487501 3075 raft_consensus.cc:504] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.487854 3075 leader_election.cc:288] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873), d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549)
I20260619 11:59:49.491406 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.488197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36873 (local address 127.0.203.194:48015)
0619 11:59:49.488258 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.488270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.488425 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.488695 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.488696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.488699 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:49.488809 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.488811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.489533 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.489535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.490446 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.490448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.491103 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.491104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.491188 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.491192 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.491270 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.491317 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 11:59:49.491406 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.488353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:48015 (local address 127.0.203.195:36873)
0619 11:59:49.488422 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.488424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.488435 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:59:49.488494 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.488495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.488508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.488604 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:49.488909 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.489432 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.490546 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.491014 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.491104 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.491115 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.491278 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.491319 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:59:49.491851 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.488478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40549 (local address 127.0.203.194:37069)
0619 11:59:49.488766 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.488776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.488848 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.489012 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.489013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.489015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:49.489101 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.489103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.489815 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.489816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.490859 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.490862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.491520 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.491522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.491600 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.491604 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.491691 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.491736 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":229,"threads_started":1}
I20260619 11:59:49.491863 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.488565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:37069 (local address 127.0.203.193:40549)
0619 11:59:49.488616 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.488619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.488833 (+ 214us) server_negotiation.cc:407] Connection header received
0619 11:59:49.488935 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.488936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.488946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.489000 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:49.489170 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.489728 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.490958 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.491413 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.491500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.491510 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.491713 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.491796 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 11:59:49.491879 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68aabdc777a54959becc07266aad93c8" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422" is_pre_election: true
I20260619 11:59:49.492066 2999 raft_consensus.cc:2454] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:49.492344 2793 leader_election.cc:302] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, ded06fb1ef014f40ba802be657332422; no voters:
I20260619 11:59:49.492499 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68aabdc777a54959becc07266aad93c8" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc" is_pre_election: true
I20260619 11:59:49.492565 3075 raft_consensus.cc:2781] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:49.492692 2719 raft_consensus.cc:2454] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:49.492725 3075 raft_consensus.cc:482] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:49.492760 3075 raft_consensus.cc:3037] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.493621 3075 raft_consensus.cc:504] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.493801 3075 leader_election.cc:288] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Requested vote from peers ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873), d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549)
I20260619 11:59:49.494118 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68aabdc777a54959becc07266aad93c8" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422"
I20260619 11:59:49.494118 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68aabdc777a54959becc07266aad93c8" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc"
I20260619 11:59:49.494223 2719 raft_consensus.cc:3037] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.494223 2999 raft_consensus.cc:3037] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.495287 2999 raft_consensus.cc:2454] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:49.495287 2719 raft_consensus.cc:2454] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:49.495551 2793 leader_election.cc:302] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, d2643e7f8e474732a6198382b99c02dc; no voters:
I20260619 11:59:49.495674 3075 raft_consensus.cc:2781] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:49.495846 3075 raft_consensus.cc:686] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 1 LEADER]: Becoming Leader. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Running, Role: LEADER
I20260619 11:59:49.495935 3075 consensus_queue.cc:237] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } }
I20260619 11:59:49.497077 2561 catalog_manager.cc:5495] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 628cfe85868b49ab8616017ae22431bc (127.0.203.194). New cstate: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } health_report { overall_health: HEALTHY } } }
Jun 19 11:59:49 dist-test-slave-fbq8 krb5kdc[2517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:49.534778 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.525653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36401 (local address 127.0.0.1:46788)
0619 11:59:49.525939 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.525952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.526113 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.526285 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.526286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.526668 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:49.526725 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.526727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.527446 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.527447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.528232 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.528239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.528805 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.528806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.528867 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.529735 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:49.529740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:49.531519 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:49.533314 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.533323 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.533325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.533610 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.533886 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:49.533887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:49.533888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:49.533956 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:49.534326 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:49.534334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:49.534394 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.534541 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.534697 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":162,"threads_started":1}
I20260619 11:59:49.535089 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.525771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.0.203.254:36401)
0619 11:59:49.525919 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.525922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.526087 (+ 165us) server_negotiation.cc:407] Connection header received
0619 11:59:49.526191 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.526191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.526209 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.526280 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:49.526819 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.527354 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.528427 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.528698 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.528807 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.531636 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:49.531638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:49.531638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:49.531645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:49.533164 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.533724 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.533726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.533726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.533783 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:49.534055 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:49.534056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:49.534057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:49.534206 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:49.534289 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.534633 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.534922 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":105}
I20260619 11:59:49.538159 2561 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:46788:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:59:49.538398 2561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:49.541527 2699 tablet_service.cc:1467] Processing CreateTablet for tablet 44892ca926744b2586b84f3222421501 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21e94c3bf37d425da697facc5416e279]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:59:49.541560 2979 tablet_service.cc:1467] Processing CreateTablet for tablet 44892ca926744b2586b84f3222421501 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21e94c3bf37d425da697facc5416e279]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:59:49.541743 2699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44892ca926744b2586b84f3222421501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.541743 2979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44892ca926744b2586b84f3222421501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.541797 2832 tablet_service.cc:1467] Processing CreateTablet for tablet 44892ca926744b2586b84f3222421501 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21e94c3bf37d425da697facc5416e279]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:59:49.542073 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44892ca926744b2586b84f3222421501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.543465 3068 tablet_bootstrap.cc:492] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: Bootstrap starting.
I20260619 11:59:49.543746 3070 tablet_bootstrap.cc:492] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: Bootstrap starting.
I20260619 11:59:49.544082 3068 tablet_bootstrap.cc:654] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.544229 3070 tablet_bootstrap.cc:654] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.544857 3068 tablet_bootstrap.cc:492] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: No bootstrap required, opened a new log
I20260619 11:59:49.544922 3068 ts_tablet_manager.cc:1397] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:49.545085 3068 raft_consensus.cc:348] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.545181 3068 raft_consensus.cc:374] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.545205 3068 raft_consensus.cc:729] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded06fb1ef014f40ba802be657332422, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.545256 3068 consensus_queue.cc:260] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.545289 3071 tablet_bootstrap.cc:492] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: Bootstrap starting.
I20260619 11:59:49.545362 3068 ts_tablet_manager.cc:1428] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 11:59:49.545532 3074 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.545593 3074 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:49.545840 3071 tablet_bootstrap.cc:654] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.546507 3070 tablet_bootstrap.cc:492] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: No bootstrap required, opened a new log
I20260619 11:59:49.546588 3070 ts_tablet_manager.cc:1397] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 11:59:49.546773 3071 tablet_bootstrap.cc:492] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: No bootstrap required, opened a new log
I20260619 11:59:49.546840 3071 ts_tablet_manager.cc:1397] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:59:49.546940 3070 raft_consensus.cc:348] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.547034 3070 raft_consensus.cc:374] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.547051 3070 raft_consensus.cc:729] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2643e7f8e474732a6198382b99c02dc, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.547061 3071 raft_consensus.cc:348] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.547145 3071 raft_consensus.cc:374] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.547118 3070 consensus_queue.cc:260] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.547165 3071 raft_consensus.cc:729] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.547212 3071 consensus_queue.cc:260] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.547243 3070 ts_tablet_manager.cc:1428] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 11:59:49.547302 3071 ts_tablet_manager.cc:1428] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:49.547421 3076 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.547515 3076 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:49.547763 3075 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.547866 3075 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:49.553998 3075 raft_consensus.cc:482] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:49.554093 3075 raft_consensus.cc:504] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.554271 3075 leader_election.cc:288] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873)
I20260619 11:59:49.554545 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44892ca926744b2586b84f3222421501" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc" is_pre_election: true
I20260619 11:59:49.554711 2719 raft_consensus.cc:2454] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:49.554672 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44892ca926744b2586b84f3222421501" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422" is_pre_election: true
I20260619 11:59:49.554767 2999 raft_consensus.cc:2454] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:49.554944 2793 leader_election.cc:302] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, d2643e7f8e474732a6198382b99c02dc; no voters:
I20260619 11:59:49.555043 3075 raft_consensus.cc:2781] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:49.555076 3075 raft_consensus.cc:482] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:49.555096 3075 raft_consensus.cc:3037] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.556043 3075 raft_consensus.cc:504] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.556205 3075 leader_election.cc:288] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Requested vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873)
I20260619 11:59:49.556417 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44892ca926744b2586b84f3222421501" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422"
I20260619 11:59:49.556519 2999 raft_consensus.cc:3037] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.556821 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44892ca926744b2586b84f3222421501" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc"
I20260619 11:59:49.556962 2719 raft_consensus.cc:3037] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.557329 2999 raft_consensus.cc:2454] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:49.557507 2793 leader_election.cc:302] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, ded06fb1ef014f40ba802be657332422; no voters:
I20260619 11:59:49.557613 3075 raft_consensus.cc:2781] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:49.557667 3075 raft_consensus.cc:686] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [term 1 LEADER]: Becoming Leader. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Running, Role: LEADER
I20260619 11:59:49.557828 3075 consensus_queue.cc:237] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.557924 2719 raft_consensus.cc:2454] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:49.558333 3078 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: TxnStatusTablet state changed. Reason: New leader 628cfe85868b49ab8616017ae22431bc. Latest consensus state: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.558471 3078 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:59:49.558681 2561 catalog_manager.cc:5495] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 628cfe85868b49ab8616017ae22431bc (127.0.203.194). New cstate: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } health_report { overall_health: UNKNOWN } } }
I20260619 11:59:49.558768 3090 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:59:49.558908 3090 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:59:49.571321 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.567349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40461 (local address 127.0.0.1:52494)
0619 11:59:49.567544 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.567560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.567748 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.568238 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.568239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.568241 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:59:49.568313 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.568316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.569410 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.569413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.570220 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.570225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.570681 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.570682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.570762 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.570780 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:59:49.571112 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:59:49.571122 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.571175 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.571242 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 11:59:49.571309 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.567585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.203.194:40461)
0619 11:59:49.567760 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.567762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.567778 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:59:49.567983 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.567985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.568102 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.568213 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:49.568405 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.569134 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.570359 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.570587 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.570679 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.570901 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:49.571035 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:49.571140 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.571181 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.571207 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 11:59:49.574502 2999 raft_consensus.cc:1261] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:49.575110 2719 raft_consensus.cc:1261] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:49.575486 3078 consensus_queue.cc:1035] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:49.575776 3075 consensus_queue.cc:1035] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:49.577913 3076 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: TxnStatusTablet state changed. Reason: New leader 628cfe85868b49ab8616017ae22431bc. Latest consensus state: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.578038 3076 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:49.578712 3074 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: TxnStatusTablet state changed. Reason: New leader 628cfe85868b49ab8616017ae22431bc. Latest consensus state: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.579088 3074 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:49.580117 3075 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.580225 3075 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:59:49.581480 3078 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.581588 3078 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P 628cfe85868b49ab8616017ae22431bc: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:59:49.582297 3076 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.582423 3076 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P d2643e7f8e474732a6198382b99c02dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:49.582859 3074 tablet_replica.cc:439] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } } }
I20260619 11:59:49.583002 3074 tablet_replica.cc:442] T 44892ca926744b2586b84f3222421501 P ded06fb1ef014f40ba802be657332422: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:59:49.583094 3093 mvcc.cc:205] Tried to move back new op lower bound from 7298541115693137920 to 7298541115629334528. Current Snapshot: MvccSnapshot[applied={T|T < 7298541115693137920 or (T in {7298541115693137920})}]
I20260619 11:59:49.586561 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.583617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.0.203.194:40461)
0619 11:59:49.583676 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.583679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.583719 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:59:49.583741 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.583741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.583752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.583792 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:49.583998 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.584494 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.585715 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.585917 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.585986 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.586254 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:49.586323 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:49.586389 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.586467 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.586499 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 11:59:49.590535 3046 tablet.cc:2369] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:59:49.594151 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.590536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40461 (local address 127.0.203.194:46325)
0619 11:59:49.590690 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:49.590808 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:49.590901 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:49.591114 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:49.591115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:49.591117 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:59:49.591182 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.591184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.592193 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.592194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.592949 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:49.592952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.593485 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.593485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:49.593545 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.593564 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:59:49.593939 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:59:49.593943 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:59:49.594002 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 11:59:49.594060 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 11:59:49.594159 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:49.590819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:46325 (local address 127.0.203.194:40461)
0619 11:59:49.590858 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:49.590860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:49.590892 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:59:49.591033 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:49.591034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:49.591045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:49.591104 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:49.591278 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.592062 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.593044 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:49.593351 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:49.593433 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:49.593704 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:49.593738 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:49.593870 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:49.594030 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 11:59:49.594051 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 11:59:49.597892 2999 raft_consensus.cc:1261] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:49.598008 2719 raft_consensus.cc:1261] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:49.598104 3078 consensus_queue.cc:1035] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:49.598306 3078 consensus_queue.cc:1035] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:49.600265 3095 mvcc.cc:205] Tried to move back new op lower bound from 7298541115791065088 to 7298541115376140288. Current Snapshot: MvccSnapshot[applied={T|T < 7298541115791065088}]
I20260619 11:59:49.601862 3094 mvcc.cc:205] Tried to move back new op lower bound from 7298541115791065088 to 7298541115376140288. Current Snapshot: MvccSnapshot[applied={T|T < 7298541115791065088}]
I20260619 11:59:49.614055 2561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:49.614198 2561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:49.615119 2561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Sending DeleteTablet for 3 replicas of tablet 68aabdc777a54959becc07266aad93c8
I20260619 11:59:49.615630 2979 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aabdc777a54959becc07266aad93c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:49 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 11:59:49.615630 2699 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aabdc777a54959becc07266aad93c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:49 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260619 11:59:49.615847 2561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:59:49.616024 2561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:49.616024 3108 tablet_replica.cc:330] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: stopping tablet replica
I20260619 11:59:49.616046 2832 tablet_service.cc:1514] Processing DeleteTablet for tablet 68aabdc777a54959becc07266aad93c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:49 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260619 11:59:49.616148 3108 raft_consensus.cc:2229] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:49.616231 3108 raft_consensus.cc:2258] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:49.616271 3110 tablet_replica.cc:330] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: stopping tablet replica
I20260619 11:59:49.616369 3110 raft_consensus.cc:2229] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:59:49.616395 3109 tablet_replica.cc:330] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: stopping tablet replica
I20260619 11:59:49.616537 3110 raft_consensus.cc:2258] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:49.616541 3109 raft_consensus.cc:2229] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:49.616600 3109 raft_consensus.cc:2258] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:49.617057 3110 ts_tablet_manager.cc:1905] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:49.618111 2832 tablet_service.cc:1467] Processing CreateTablet for tablet fcf1ab37135640dda2ec049d1c864bf4 (DEFAULT_TABLE table=test-table [id=4d07fbc27623422b8138a1d65f064dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:49.618158 2979 tablet_service.cc:1467] Processing CreateTablet for tablet fcf1ab37135640dda2ec049d1c864bf4 (DEFAULT_TABLE table=test-table [id=4d07fbc27623422b8138a1d65f064dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:49.618283 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf1ab37135640dda2ec049d1c864bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.618306 2979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf1ab37135640dda2ec049d1c864bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.618394 3109 ts_tablet_manager.cc:1905] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:49.619875 3068 tablet_bootstrap.cc:492] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: Bootstrap starting.
I20260619 11:59:49.620347 3068 tablet_bootstrap.cc:654] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.620339 2699 tablet_service.cc:1467] Processing CreateTablet for tablet fcf1ab37135640dda2ec049d1c864bf4 (DEFAULT_TABLE table=test-table [id=4d07fbc27623422b8138a1d65f064dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:49.620457 2699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf1ab37135640dda2ec049d1c864bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:49.620739 3071 tablet_bootstrap.cc:492] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Bootstrap starting.
I20260619 11:59:49.621236 3071 tablet_bootstrap.cc:654] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.621785 3070 tablet_bootstrap.cc:492] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: Bootstrap starting.
I20260619 11:59:49.622285 3070 tablet_bootstrap.cc:654] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:49.622499 3068 tablet_bootstrap.cc:492] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: No bootstrap required, opened a new log
I20260619 11:59:49.622550 3068 ts_tablet_manager.cc:1397] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 11:59:49.622711 3068 raft_consensus.cc:348] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.622799 3068 raft_consensus.cc:374] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.622825 3068 raft_consensus.cc:729] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded06fb1ef014f40ba802be657332422, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.622871 3068 consensus_queue.cc:260] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.622962 3068 ts_tablet_manager.cc:1428] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 11:59:49.622996 3070 tablet_bootstrap.cc:492] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: No bootstrap required, opened a new log
I20260619 11:59:49.623041 3070 ts_tablet_manager.cc:1397] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:49.623220 3070 raft_consensus.cc:348] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.623312 3070 raft_consensus.cc:374] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.623337 3070 raft_consensus.cc:729] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2643e7f8e474732a6198382b99c02dc, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.623389 3070 consensus_queue.cc:260] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.623492 3070 ts_tablet_manager.cc:1428] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:49.624182 3071 tablet_bootstrap.cc:492] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: No bootstrap required, opened a new log
I20260619 11:59:49.624243 3071 ts_tablet_manager.cc:1397] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 11:59:49.624280 3110 ts_tablet_manager.cc:1918] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:49.624359 3110 log.cc:1196] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/68aabdc777a54959becc07266aad93c8
I20260619 11:59:49.624420 3071 raft_consensus.cc:348] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.624508 3071 raft_consensus.cc:374] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:49.624533 3071 raft_consensus.cc:729] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Initialized, Role: FOLLOWER
I20260619 11:59:49.624593 3071 consensus_queue.cc:260] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.624725 3110 ts_tablet_manager.cc:1939] T 68aabdc777a54959becc07266aad93c8 P 628cfe85868b49ab8616017ae22431bc: Deleting consensus metadata
I20260619 11:59:49.624857 3109 ts_tablet_manager.cc:1918] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:49.624920 3109 log.cc:1196] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/68aabdc777a54959becc07266aad93c8
I20260619 11:59:49.625166 3109 ts_tablet_manager.cc:1939] T 68aabdc777a54959becc07266aad93c8 P d2643e7f8e474732a6198382b99c02dc: Deleting consensus metadata
I20260619 11:59:49.625317 2549 catalog_manager.cc:4841] TS 628cfe85868b49ab8616017ae22431bc (127.0.203.194:40461): tablet 68aabdc777a54959becc07266aad93c8 (table test-table [id=f23b867baa0d4fc29115ff62cd10841b]) successfully deleted
I20260619 11:59:49.625773 2549 catalog_manager.cc:4841] TS d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549): tablet 68aabdc777a54959becc07266aad93c8 (table test-table [id=f23b867baa0d4fc29115ff62cd10841b]) successfully deleted
I20260619 11:59:49.624712 3071 ts_tablet_manager.cc:1428] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:49.629040 3108 ts_tablet_manager.cc:1905] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:49.630795 3108 ts_tablet_manager.cc:1918] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:49.630898 3108 log.cc:1196] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/68aabdc777a54959becc07266aad93c8
I20260619 11:59:49.631153 3108 ts_tablet_manager.cc:1939] T 68aabdc777a54959becc07266aad93c8 P ded06fb1ef014f40ba802be657332422: Deleting consensus metadata
I20260619 11:59:49.631659 2549 catalog_manager.cc:4841] TS ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873): tablet 68aabdc777a54959becc07266aad93c8 (table test-table [id=f23b867baa0d4fc29115ff62cd10841b]) successfully deleted
W20260619 11:59:49.644794 2906 tablet.cc:2369] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:59:49.656849 3075 raft_consensus.cc:482] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:49.656962 3075 raft_consensus.cc:504] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.657127 3075 leader_election.cc:288] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873)
I20260619 11:59:49.657618 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf1ab37135640dda2ec049d1c864bf4" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422" is_pre_election: true
I20260619 11:59:49.657658 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf1ab37135640dda2ec049d1c864bf4" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc" is_pre_election: true
I20260619 11:59:49.657783 2719 raft_consensus.cc:2454] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:49.657783 2999 raft_consensus.cc:2454] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:49.658006 2793 leader_election.cc:302] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, d2643e7f8e474732a6198382b99c02dc; no voters:
I20260619 11:59:49.658155 3075 raft_consensus.cc:2781] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:49.658209 3075 raft_consensus.cc:482] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:49.658228 3075 raft_consensus.cc:3037] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.659227 3075 raft_consensus.cc:504] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.659409 3075 leader_election.cc:288] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Requested vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873)
I20260619 11:59:49.659621 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf1ab37135640dda2ec049d1c864bf4" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422"
I20260619 11:59:49.659621 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf1ab37135640dda2ec049d1c864bf4" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc"
I20260619 11:59:49.659694 2999 raft_consensus.cc:3037] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.659695 2719 raft_consensus.cc:3037] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:49.660559 2999 raft_consensus.cc:2454] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:49.660559 2719 raft_consensus.cc:2454] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:49.660784 2793 leader_election.cc:302] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, ded06fb1ef014f40ba802be657332422; no voters:
I20260619 11:59:49.660928 3075 raft_consensus.cc:2781] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:49.661051 3075 raft_consensus.cc:686] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 1 LEADER]: Becoming Leader. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Running, Role: LEADER
I20260619 11:59:49.661108 3075 consensus_queue.cc:237] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:49.661914 2560 catalog_manager.cc:5495] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 628cfe85868b49ab8616017ae22431bc (127.0.203.194). New cstate: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } health_report { overall_health: UNKNOWN } } }
I20260619 11:59:49.677515 2719 raft_consensus.cc:1261] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:49.677527 2999 raft_consensus.cc:1261] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:49.677834 3078 consensus_queue.cc:1035] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:49.678036 3075 consensus_queue.cc:1035] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:59:49.683943 3093 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:59:51.209554 2560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:51.209705 2560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:51.210924 2560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Sending DeleteTablet for 3 replicas of tablet fcf1ab37135640dda2ec049d1c864bf4
I20260619 11:59:51.211225 2699 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf1ab37135640dda2ec049d1c864bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:51 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260619 11:59:51.211395 2832 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf1ab37135640dda2ec049d1c864bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:51 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260619 11:59:51.211587 2979 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf1ab37135640dda2ec049d1c864bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:51 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 11:59:51.211619 3136 tablet_replica.cc:330] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: stopping tablet replica
I20260619 11:59:51.211670 3136 raft_consensus.cc:2229] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:51.211712 3136 raft_consensus.cc:2258] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:51.211925 3137 tablet_replica.cc:330] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: stopping tablet replica
I20260619 11:59:51.212038 3136 ts_tablet_manager.cc:1905] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:51.212038 3138 tablet_replica.cc:330] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: stopping tablet replica
I20260619 11:59:51.212145 3137 raft_consensus.cc:2229] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:59:51.212241 3138 raft_consensus.cc:2229] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:51.212363 3137 raft_consensus.cc:2258] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:51.212445 3138 raft_consensus.cc:2258] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:51.212800 3138 ts_tablet_manager.cc:1905] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:51.214200 3137 ts_tablet_manager.cc:1905] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:51.214540 3138 ts_tablet_manager.cc:1918] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:59:51.214605 3138 log.cc:1196] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/fcf1ab37135640dda2ec049d1c864bf4
I20260619 11:59:51.214821 3138 ts_tablet_manager.cc:1939] T fcf1ab37135640dda2ec049d1c864bf4 P ded06fb1ef014f40ba802be657332422: Deleting consensus metadata
I20260619 11:59:51.215287 2549 catalog_manager.cc:4841] TS ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873): tablet fcf1ab37135640dda2ec049d1c864bf4 (table test-table [id=4d07fbc27623422b8138a1d65f064dfe]) successfully deleted
I20260619 11:59:51.215844 3136 ts_tablet_manager.cc:1918] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:59:51.215898 3136 log.cc:1196] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/fcf1ab37135640dda2ec049d1c864bf4
I20260619 11:59:51.216049 3137 ts_tablet_manager.cc:1918] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:59:51.216113 3137 log.cc:1196] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/fcf1ab37135640dda2ec049d1c864bf4
I20260619 11:59:51.216153 3136 ts_tablet_manager.cc:1939] T fcf1ab37135640dda2ec049d1c864bf4 P d2643e7f8e474732a6198382b99c02dc: Deleting consensus metadata
I20260619 11:59:51.216364 3137 ts_tablet_manager.cc:1939] T fcf1ab37135640dda2ec049d1c864bf4 P 628cfe85868b49ab8616017ae22431bc: Deleting consensus metadata
I20260619 11:59:51.216674 2549 catalog_manager.cc:4841] TS d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549): tablet fcf1ab37135640dda2ec049d1c864bf4 (table test-table [id=4d07fbc27623422b8138a1d65f064dfe]) successfully deleted
I20260619 11:59:51.216810 2549 catalog_manager.cc:4841] TS 628cfe85868b49ab8616017ae22431bc (127.0.203.194:40461): tablet fcf1ab37135640dda2ec049d1c864bf4 (table test-table [id=4d07fbc27623422b8138a1d65f064dfe]) successfully deleted
Jun 19 11:59:51 dist-test-slave-fbq8 krb5kdc[2517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:59:51 dist-test-slave-fbq8 krb5kdc[2517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:51.246794 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.232789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.0.203.254:36401)
0619 11:59:51.232992 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:51.232994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:51.233114 (+ 120us) server_negotiation.cc:407] Connection header received
0619 11:59:51.233155 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:51.233156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:51.233170 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:51.233271 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:51.233986 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.234548 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.235405 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.235616 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.235682 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.243419 (+ 7737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:51.243447 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:51.243447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:51.243457 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:51.245330 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:51.245841 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:51.245842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:51.245843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:51.245905 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:51.246177 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:51.246177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:51.246178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:51.246326 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:51.246414 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:51.246634 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 11:59:51.246718 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":136,"threads_started":1}
I20260619 11:59:51.249017 2560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:59:51.249190 2560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:51.251129 2699 tablet_service.cc:1467] Processing CreateTablet for tablet fcadca7ecb404acf89a4acb5d94055d7 (DEFAULT_TABLE table=test-table [id=3d17284e0b33496c97e06984b96cb2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:51.251125 2832 tablet_service.cc:1467] Processing CreateTablet for tablet fcadca7ecb404acf89a4acb5d94055d7 (DEFAULT_TABLE table=test-table [id=3d17284e0b33496c97e06984b96cb2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:51.251176 2979 tablet_service.cc:1467] Processing CreateTablet for tablet fcadca7ecb404acf89a4acb5d94055d7 (DEFAULT_TABLE table=test-table [id=3d17284e0b33496c97e06984b96cb2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:51.251317 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcadca7ecb404acf89a4acb5d94055d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:51.251317 2979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcadca7ecb404acf89a4acb5d94055d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:51.251345 2699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcadca7ecb404acf89a4acb5d94055d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:51.252736 3147 tablet_bootstrap.cc:492] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: Bootstrap starting.
I20260619 11:59:51.253054 3148 tablet_bootstrap.cc:492] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: Bootstrap starting.
I20260619 11:59:51.253135 3147 tablet_bootstrap.cc:654] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:51.253217 3149 tablet_bootstrap.cc:492] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: Bootstrap starting.
I20260619 11:59:51.253585 3148 tablet_bootstrap.cc:654] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:51.253726 3149 tablet_bootstrap.cc:654] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:51.253793 3147 tablet_bootstrap.cc:492] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: No bootstrap required, opened a new log
I20260619 11:59:51.253871 3147 ts_tablet_manager.cc:1397] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:51.254038 3147 raft_consensus.cc:348] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.254134 3147 raft_consensus.cc:374] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:51.254161 3147 raft_consensus.cc:729] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2643e7f8e474732a6198382b99c02dc, State: Initialized, Role: FOLLOWER
I20260619 11:59:51.254242 3147 consensus_queue.cc:260] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.254364 3148 tablet_bootstrap.cc:492] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: No bootstrap required, opened a new log
I20260619 11:59:51.254444 3148 ts_tablet_manager.cc:1397] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:51.254457 3147 ts_tablet_manager.cc:1428] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:51.254614 3148 raft_consensus.cc:348] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.254700 3148 raft_consensus.cc:374] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:51.254721 3148 raft_consensus.cc:729] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Initialized, Role: FOLLOWER
I20260619 11:59:51.254772 3148 consensus_queue.cc:260] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.254896 3148 ts_tablet_manager.cc:1428] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:51.255309 3149 tablet_bootstrap.cc:492] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: No bootstrap required, opened a new log
I20260619 11:59:51.255384 3149 ts_tablet_manager.cc:1397] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:59:51.255548 3149 raft_consensus.cc:348] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.255638 3149 raft_consensus.cc:374] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:51.255661 3149 raft_consensus.cc:729] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded06fb1ef014f40ba802be657332422, State: Initialized, Role: FOLLOWER
I20260619 11:59:51.255708 3149 consensus_queue.cc:260] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.255899 3149 ts_tablet_manager.cc:1428] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 11:59:51.428906 3078 raft_consensus.cc:482] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:51.429073 3078 raft_consensus.cc:504] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.429304 3078 leader_election.cc:288] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873)
I20260619 11:59:51.429569 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcadca7ecb404acf89a4acb5d94055d7" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc" is_pre_election: true
I20260619 11:59:51.429703 2719 raft_consensus.cc:2454] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:51.429656 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcadca7ecb404acf89a4acb5d94055d7" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422" is_pre_election: true
I20260619 11:59:51.429746 2999 raft_consensus.cc:2454] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 0.
I20260619 11:59:51.429919 2793 leader_election.cc:302] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, ded06fb1ef014f40ba802be657332422; no voters:
I20260619 11:59:51.430013 3078 raft_consensus.cc:2781] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:51.430051 3078 raft_consensus.cc:482] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:51.430073 3078 raft_consensus.cc:3037] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:51.430838 3078 raft_consensus.cc:504] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.430998 3078 leader_election.cc:288] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Requested vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873)
I20260619 11:59:51.431354 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcadca7ecb404acf89a4acb5d94055d7" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc"
I20260619 11:59:51.431370 2999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcadca7ecb404acf89a4acb5d94055d7" candidate_uuid: "628cfe85868b49ab8616017ae22431bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded06fb1ef014f40ba802be657332422"
I20260619 11:59:51.431478 2719 raft_consensus.cc:3037] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:51.431481 2999 raft_consensus.cc:3037] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:51.432228 2719 raft_consensus.cc:2454] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:51.432447 2999 raft_consensus.cc:2454] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 628cfe85868b49ab8616017ae22431bc in term 1.
I20260619 11:59:51.432474 2793 leader_election.cc:302] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 628cfe85868b49ab8616017ae22431bc, d2643e7f8e474732a6198382b99c02dc; no voters:
I20260619 11:59:51.432591 3078 raft_consensus.cc:2781] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:51.432672 3078 raft_consensus.cc:686] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 1 LEADER]: Becoming Leader. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Running, Role: LEADER
I20260619 11:59:51.432727 3078 consensus_queue.cc:237] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.433629 2562 catalog_manager.cc:5495] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 628cfe85868b49ab8616017ae22431bc (127.0.203.194). New cstate: current_term: 1 leader_uuid: "628cfe85868b49ab8616017ae22431bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } health_report { overall_health: UNKNOWN } } }
I20260619 11:59:51.443692 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.440666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.0.203.194:40461)
0619 11:59:51.440888 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:51.440890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:51.440902 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:59:51.440953 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:51.440954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:51.440967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:51.441085 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:51.441264 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.441808 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.442912 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.443145 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.443231 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.443406 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:51.443443 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:51.443505 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:51.443568 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:59:51.443589 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
I20260619 11:59:51.446363 2999 raft_consensus.cc:1261] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:51.446388 2719 raft_consensus.cc:1261] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Refusing update from remote peer 628cfe85868b49ab8616017ae22431bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:51.446666 3159 consensus_queue.cc:1035] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:51.446806 3124 consensus_queue.cc:1035] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:51.458220 2562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:51.458334 2562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:51.459244 2562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Sending DeleteTablet for 3 replicas of tablet fcadca7ecb404acf89a4acb5d94055d7
I20260619 11:59:51.459555 2832 tablet_service.cc:1514] Processing DeleteTablet for tablet fcadca7ecb404acf89a4acb5d94055d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:51 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260619 11:59:51.459546 2699 tablet_service.cc:1514] Processing DeleteTablet for tablet fcadca7ecb404acf89a4acb5d94055d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:51 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260619 11:59:51.459574 2979 tablet_service.cc:1514] Processing DeleteTablet for tablet fcadca7ecb404acf89a4acb5d94055d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:51 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 11:59:51.459646 3137 tablet_replica.cc:330] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: stopping tablet replica
I20260619 11:59:51.459646 3136 tablet_replica.cc:330] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: stopping tablet replica
I20260619 11:59:51.459690 3136 raft_consensus.cc:2229] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:51.459692 3137 raft_consensus.cc:2229] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:59:51.459728 3136 raft_consensus.cc:2258] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:51.459750 3137 raft_consensus.cc:2258] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:51.459823 3138 tablet_replica.cc:330] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: stopping tablet replica
I20260619 11:59:51.459879 3138 raft_consensus.cc:2229] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:51.459926 3138 raft_consensus.cc:2258] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:51.460088 3136 ts_tablet_manager.cc:1905] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:51.460116 2562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:59:51.460295 2562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:59:51.460337 3138 ts_tablet_manager.cc:1905] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:51.461930 3137 ts_tablet_manager.cc:1905] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:51.461956 3138 ts_tablet_manager.cc:1918] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:51.461990 3136 ts_tablet_manager.cc:1918] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:51.462013 3138 log.cc:1196] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/fcadca7ecb404acf89a4acb5d94055d7
I20260619 11:59:51.462036 3136 log.cc:1196] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/fcadca7ecb404acf89a4acb5d94055d7
I20260619 11:59:51.462268 3136 ts_tablet_manager.cc:1939] T fcadca7ecb404acf89a4acb5d94055d7 P d2643e7f8e474732a6198382b99c02dc: Deleting consensus metadata
I20260619 11:59:51.462268 3138 ts_tablet_manager.cc:1939] T fcadca7ecb404acf89a4acb5d94055d7 P ded06fb1ef014f40ba802be657332422: Deleting consensus metadata
I20260619 11:59:51.462867 2549 catalog_manager.cc:4841] TS ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873): tablet fcadca7ecb404acf89a4acb5d94055d7 (table test-table [id=3d17284e0b33496c97e06984b96cb2e0]) successfully deleted
I20260619 11:59:51.463294 2832 tablet_service.cc:1467] Processing CreateTablet for tablet efdfbd54ce084574864326ed3f2f7f31 (DEFAULT_TABLE table=test-table [id=80e5949081364f1d9baa6ab0150fd5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:51.463404 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdfbd54ce084574864326ed3f2f7f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:51.463320 2979 tablet_service.cc:1467] Processing CreateTablet for tablet efdfbd54ce084574864326ed3f2f7f31 (DEFAULT_TABLE table=test-table [id=80e5949081364f1d9baa6ab0150fd5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:51.463546 2979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdfbd54ce084574864326ed3f2f7f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:51.464299 3137 ts_tablet_manager.cc:1918] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:59:51.464448 3137 log.cc:1196] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/fcadca7ecb404acf89a4acb5d94055d7
I20260619 11:59:51.464807 3137 ts_tablet_manager.cc:1939] T fcadca7ecb404acf89a4acb5d94055d7 P 628cfe85868b49ab8616017ae22431bc: Deleting consensus metadata
I20260619 11:59:51.464869 2549 catalog_manager.cc:4841] TS d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549): tablet fcadca7ecb404acf89a4acb5d94055d7 (table test-table [id=3d17284e0b33496c97e06984b96cb2e0]) successfully deleted
I20260619 11:59:51.465008 2699 tablet_service.cc:1467] Processing CreateTablet for tablet efdfbd54ce084574864326ed3f2f7f31 (DEFAULT_TABLE table=test-table [id=80e5949081364f1d9baa6ab0150fd5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:59:51.465109 2699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdfbd54ce084574864326ed3f2f7f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:51.465333 3149 tablet_bootstrap.cc:492] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: Bootstrap starting.
I20260619 11:59:51.465368 3148 tablet_bootstrap.cc:492] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc: Bootstrap starting.
I20260619 11:59:51.465749 3148 tablet_bootstrap.cc:654] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:51.465749 3149 tablet_bootstrap.cc:654] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:51.465970 3147 tablet_bootstrap.cc:492] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc: Bootstrap starting.
I20260619 11:59:51.466179 2549 catalog_manager.cc:4841] TS 628cfe85868b49ab8616017ae22431bc (127.0.203.194:40461): tablet fcadca7ecb404acf89a4acb5d94055d7 (table test-table [id=3d17284e0b33496c97e06984b96cb2e0]) successfully deleted
I20260619 11:59:51.466588 3148 tablet_bootstrap.cc:492] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc: No bootstrap required, opened a new log
I20260619 11:59:51.466668 3147 tablet_bootstrap.cc:654] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:51.466677 3148 ts_tablet_manager.cc:1397] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:51.466814 3148 raft_consensus.cc:348] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.466874 3148 raft_consensus.cc:374] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:51.466863 3149 tablet_bootstrap.cc:492] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: No bootstrap required, opened a new log
I20260619 11:59:51.466890 3148 raft_consensus.cc:729] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628cfe85868b49ab8616017ae22431bc, State: Initialized, Role: FOLLOWER
I20260619 11:59:51.466919 3149 ts_tablet_manager.cc:1397] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:59:51.466952 3148 consensus_queue.cc:260] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.467051 3148 ts_tablet_manager.cc:1428] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 11:59:51.467099 3149 raft_consensus.cc:348] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.467193 3149 raft_consensus.cc:374] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:51.467245 3149 raft_consensus.cc:729] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded06fb1ef014f40ba802be657332422, State: Initialized, Role: FOLLOWER
I20260619 11:59:51.467304 3149 consensus_queue.cc:260] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.467403 3149 ts_tablet_manager.cc:1428] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 11:59:51.467541 3147 tablet_bootstrap.cc:492] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc: No bootstrap required, opened a new log
I20260619 11:59:51.467609 3147 ts_tablet_manager.cc:1397] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:59:51.467793 3147 raft_consensus.cc:348] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.467891 3147 raft_consensus.cc:374] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:51.467914 3147 raft_consensus.cc:729] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2643e7f8e474732a6198382b99c02dc, State: Initialized, Role: FOLLOWER
I20260619 11:59:51.467962 3147 consensus_queue.cc:260] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.468073 3147 ts_tablet_manager.cc:1428] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:51.472772 3154 raft_consensus.cc:482] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:59:51.472848 3154 raft_consensus.cc:504] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.473176 3154 leader_election.cc:288] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), 628cfe85868b49ab8616017ae22431bc (127.0.203.194:40461)
I20260619 11:59:51.477152 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.473399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40549 (local address 127.0.203.195:33509)
0619 11:59:51.473564 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:51.473578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:51.473655 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:51.473929 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:51.473931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:51.473934 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:51.474091 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.474094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.475288 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.475289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.476131 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.476133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.476884 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.476885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.476954 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.476959 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:59:51.477024 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:59:51.477064 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":119,"threads_started":1}
I20260619 11:59:51.477170 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.473445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:33509 (local address 127.0.203.193:40549)
0619 11:59:51.473574 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:51.473576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:51.473662 (+ 86us) server_negotiation.cc:407] Connection header received
0619 11:59:51.473758 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:51.473759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:51.473773 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:51.473863 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:51.474275 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.475172 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.476226 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.476797 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.476866 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.476876 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:51.477040 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 11:59:51.477076 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260619 11:59:51.477624 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdfbd54ce084574864326ed3f2f7f31" candidate_uuid: "ded06fb1ef014f40ba802be657332422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc" is_pre_election: true
I20260619 11:59:51.477731 2719 raft_consensus.cc:2454] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded06fb1ef014f40ba802be657332422 in term 0.
I20260619 11:59:51.477955 2933 leader_election.cc:302] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2643e7f8e474732a6198382b99c02dc, ded06fb1ef014f40ba802be657332422; no voters:
I20260619 11:59:51.478152 3154 raft_consensus.cc:2781] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:59:51.478224 3154 raft_consensus.cc:482] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:59:51.478231 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.473870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:39987 (local address 127.0.203.194:40461)
0619 11:59:51.473930 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:51.473932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:51.474047 (+ 115us) server_negotiation.cc:407] Connection header received
0619 11:59:51.474214 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:51.474215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:51.474235 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:51.474291 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:59:51.474440 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.475171 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.477263 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.477840 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.477890 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.477901 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:51.478092 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 11:59:51.478157 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:59:51.478253 3154 raft_consensus.cc:3037] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:51.478259 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.473654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40461 (local address 127.0.203.195:39987)
0619 11:59:51.473958 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:51.473987 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:51.474080 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:51.474298 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:51.474299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:51.474301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:59:51.474359 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.474361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.476236 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.476236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.477141 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.477143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.477958 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.477959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.478010 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.478012 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 11:59:51.478053 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 11:59:51.478080 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":92,"threads_started":1}
I20260619 11:59:51.478660 2859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdfbd54ce084574864326ed3f2f7f31" candidate_uuid: "ded06fb1ef014f40ba802be657332422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628cfe85868b49ab8616017ae22431bc" is_pre_election: true
I20260619 11:59:51.478807 2859 raft_consensus.cc:2454] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded06fb1ef014f40ba802be657332422 in term 0.
I20260619 11:59:51.479295 3154 raft_consensus.cc:504] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.479442 3154 leader_election.cc:288] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [CANDIDATE]: Term 1 election: Requested vote from peers d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549), 628cfe85868b49ab8616017ae22431bc (127.0.203.194:40461)
I20260619 11:59:51.479611 2719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdfbd54ce084574864326ed3f2f7f31" candidate_uuid: "ded06fb1ef014f40ba802be657332422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2643e7f8e474732a6198382b99c02dc"
I20260619 11:59:51.479701 2719 raft_consensus.cc:3037] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:51.479655 2859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdfbd54ce084574864326ed3f2f7f31" candidate_uuid: "ded06fb1ef014f40ba802be657332422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628cfe85868b49ab8616017ae22431bc"
I20260619 11:59:51.479735 2859 raft_consensus.cc:3037] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:51.480520 2859 raft_consensus.cc:2454] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded06fb1ef014f40ba802be657332422 in term 1.
I20260619 11:59:51.480520 2719 raft_consensus.cc:2454] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded06fb1ef014f40ba802be657332422 in term 1.
I20260619 11:59:51.480717 2933 leader_election.cc:302] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2643e7f8e474732a6198382b99c02dc, ded06fb1ef014f40ba802be657332422; no voters:
I20260619 11:59:51.480852 3154 raft_consensus.cc:2781] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:51.481005 3154 raft_consensus.cc:686] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 1 LEADER]: Becoming Leader. State: Replica: ded06fb1ef014f40ba802be657332422, State: Running, Role: LEADER
I20260619 11:59:51.481065 3154 consensus_queue.cc:237] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } }
I20260619 11:59:51.481956 2562 catalog_manager.cc:5495] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 reported cstate change: term changed from 0 to 1, leader changed from <none> to ded06fb1ef014f40ba802be657332422 (127.0.203.195). New cstate: current_term: 1 leader_uuid: "ded06fb1ef014f40ba802be657332422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded06fb1ef014f40ba802be657332422" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36873 } health_report { overall_health: HEALTHY } } }
I20260619 11:59:51.494453 3170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.491710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.0.203.195:36873)
0619 11:59:51.491867 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:51.491868 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:51.491874 (+ 6us) server_negotiation.cc:407] Connection header received
0619 11:59:51.491908 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:51.491908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:51.491930 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:51.492007 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:51.492168 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.492780 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.493581 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.493806 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.493873 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.494039 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:51.494163 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:51.494228 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:51.494278 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 11:59:51.494297 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260619 11:59:51.500624 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.497782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:47123 (local address 127.0.203.194:40461)
0619 11:59:51.497839 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:51.497843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:51.498022 (+ 179us) server_negotiation.cc:407] Connection header received
0619 11:59:51.498050 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:51.498051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:51.498064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:51.498112 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:51.498329 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.498816 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.499723 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.499913 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.499986 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.500324 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:51.500357 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:51.500428 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:51.500536 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 11:59:51.500558 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:59:51.500826 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.497696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40461 (local address 127.0.203.195:47123)
0619 11:59:51.497891 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:51.497905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:51.497971 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:51.498148 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:51.498150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:51.498151 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:59:51.498227 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.498228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.498935 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.498936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.499541 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.499543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.500103 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.500105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.500177 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.500193 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:59:51.500468 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:59:51.500471 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:59:51.500532 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 11:59:51.500584 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
I20260619 11:59:51.503809 2859 raft_consensus.cc:1261] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Refusing update from remote peer ded06fb1ef014f40ba802be657332422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:51.504031 2719 raft_consensus.cc:1261] T efdfbd54ce084574864326ed3f2f7f31 P d2643e7f8e474732a6198382b99c02dc [term 1 FOLLOWER]: Refusing update from remote peer ded06fb1ef014f40ba802be657332422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:59:51.504161 3154 consensus_queue.cc:1035] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "628cfe85868b49ab8616017ae22431bc" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:51.504357 3168 consensus_queue.cc:1035] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2643e7f8e474732a6198382b99c02dc" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:59:51.513168 3170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.510211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:50739 (local address 127.0.203.195:36873)
0619 11:59:51.510283 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:51.510286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:51.510466 (+ 180us) server_negotiation.cc:407] Connection header received
0619 11:59:51.510500 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:51.510501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:51.510513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:51.510556 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:59:51.510759 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.511284 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.512318 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.512616 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.512750 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.512943 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:59:51.512973 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:59:51.513018 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:51.513087 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 11:59:51.513108 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:59:51.513190 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:51.510126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36873 (local address 127.0.203.194:50739)
0619 11:59:51.510334 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:51.510346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:51.510410 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:51.510571 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:51.510572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:51.510575 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:59:51.510651 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.510653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.511371 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.511403 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.512227 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:51.512229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:51.512764 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:51.512765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:51.512826 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:51.512831 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:59:51.513038 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:59:51.513040 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 11:59:51.513077 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 11:59:51.513118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":124,"threads_started":1}
W20260619 11:59:51.513765 3121 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:59:53.041778 2560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:53.041910 2560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:59:53.043159 2560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cccefb01dbf6463ba6d431ec119597ad: Sending DeleteTablet for 3 replicas of tablet efdfbd54ce084574864326ed3f2f7f31
I20260619 11:59:53.043570 2699 tablet_service.cc:1514] Processing DeleteTablet for tablet efdfbd54ce084574864326ed3f2f7f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:53 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260619 11:59:53.043640 2979 tablet_service.cc:1514] Processing DeleteTablet for tablet efdfbd54ce084574864326ed3f2f7f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:53 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 11:59:53.043685 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 2634
I20260619 11:59:53.043860 2832 tablet_service.cc:1514] Processing DeleteTablet for tablet efdfbd54ce084574864326ed3f2f7f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:59:53 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260619 11:59:53.044690 3202 tablet_replica.cc:330] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: stopping tablet replica
I20260619 11:59:53.044762 3202 raft_consensus.cc:2229] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:59:53.044888 3202 raft_consensus.cc:2258] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:53.045189 3202 ts_tablet_manager.cc:1905] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:53.047683 3202 ts_tablet_manager.cc:1918] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:59:53.047765 3202 log.cc:1196] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/efdfbd54ce084574864326ed3f2f7f31
I20260619 11:59:53.048084 3202 ts_tablet_manager.cc:1939] T efdfbd54ce084574864326ed3f2f7f31 P ded06fb1ef014f40ba802be657332422: Deleting consensus metadata
I20260619 11:59:53.048524 2549 catalog_manager.cc:4841] TS ded06fb1ef014f40ba802be657332422 (127.0.203.195:36873): tablet efdfbd54ce084574864326ed3f2f7f31 (table test-table [id=80e5949081364f1d9baa6ab0150fd5c2]) successfully deleted
I20260619 11:59:53.052198 3203 tablet_replica.cc:330] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc: stopping tablet replica
W20260619 11:59:53.052196 2549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:40549 (error 108)
I20260619 11:59:53.052309 3203 raft_consensus.cc:2229] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:59:53.052363 3203 raft_consensus.cc:2258] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:59:53.052799 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 2774
I20260619 11:59:53.052811 3203 ts_tablet_manager.cc:1905] T efdfbd54ce084574864326ed3f2f7f31 P 628cfe85868b49ab8616017ae22431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:59:53.053989 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:53.052841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40549 (local address 127.0.0.1:38978)
0619 11:59:53.053864 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:53.053927 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":109,"threads_started":1}
W20260619 11:59:53.054102 2549 catalog_manager.cc:4584] TS d2643e7f8e474732a6198382b99c02dc (127.0.203.193:40549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efdfbd54ce084574864326ed3f2f7f31: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40549: connect: Connection refused (error 111)
I20260619 11:59:53.059342 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:53.059188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40461 (local address 127.0.0.1:52516)
0619 11:59:53.059251 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:53.059307 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.194:40461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 11:59:53.059351 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 2914
I20260619 11:59:53.066815 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 2530
2026-06-19T11:59:53Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5733 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:53 dist-test-slave-fbq8 krb5kdc[3208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:53 dist-test-slave-fbq8 krb5kdc[3208](info): set up 2 sockets
Jun 19 11:59:53 dist-test-slave-fbq8 krb5kdc[3208](info): commencing operation
krb5kdc: starting...
W20260619 11:59:54.156792 815 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:54.286082 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:43069
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:54.378856 3221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:54.379091 3221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:54.379112 3221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:54.380227 3221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:54.380268 3221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:54.380282 3221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:54.380295 3221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:54.380307 3221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:54.381778 3221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:54.381981 3221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:54.382159 3221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:54.387538 3226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.387537 3229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.387537 3227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:54.387876 3221 server_base.cc:1034] running on GCE node
I20260619 11:59:54.388171 3221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:54.388489 3221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:54.389639 3221 hybrid_clock.cc:648] HybridClock initialized: now 1781870394389646 us; error 55 us; skew 500 ppm
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:54.392504 3221 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:54.393106 3221 webserver.cc:460] Webserver started at http://127.0.203.254:33403/ using document root <none> and password file <none>
I20260619 11:59:54.393366 3221 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:54.393419 3221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:54.393553 3221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:54.394721 3221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "af02755a5f334e0ca86d3724d03dfb12"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "9ed3a7f8d92ebfc9049f40b148b8b279"
server_key_iv: "9871ab222f931f007752db74da167dcc"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.395134 3221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "af02755a5f334e0ca86d3724d03dfb12"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "9ed3a7f8d92ebfc9049f40b148b8b279"
server_key_iv: "9871ab222f931f007752db74da167dcc"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.396834 3221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 11:59:54.397945 3236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.398181 3221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:54.398288 3221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "af02755a5f334e0ca86d3724d03dfb12"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "9ed3a7f8d92ebfc9049f40b148b8b279"
server_key_iv: "9871ab222f931f007752db74da167dcc"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.398375 3221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:54.407876 3221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:54.410805 3221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:54.410948 3221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:54.416198 3221 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:43069
I20260619 11:59:54.416287 3288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:43069 every 8 connection(s)
I20260619 11:59:54.416944 3221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:54.417774 3289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:54.420236 3289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12: Bootstrap starting.
I20260619 11:59:54.420980 3289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:54.421288 3289 log.cc:824] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:54.422006 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 3221
I20260619 11:59:54.422117 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:54.422258 3289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12: No bootstrap required, opened a new log
I20260619 11:59:54.422310 815 external_mini_cluster.cc:1428] Setting key b4f98dd2f30495e32eb56a9b62929853
I20260619 11:59:54.424751 3289 raft_consensus.cc:348] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af02755a5f334e0ca86d3724d03dfb12" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43069 } }
I20260619 11:59:54.424935 3289 raft_consensus.cc:374] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:54.424975 3289 raft_consensus.cc:729] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af02755a5f334e0ca86d3724d03dfb12, State: Initialized, Role: FOLLOWER
I20260619 11:59:54.425164 3289 consensus_queue.cc:260] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af02755a5f334e0ca86d3724d03dfb12" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43069 } }
I20260619 11:59:54.425269 3289 raft_consensus.cc:388] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:54.425304 3289 raft_consensus.cc:482] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:54.425350 3289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:54.426080 3289 raft_consensus.cc:504] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af02755a5f334e0ca86d3724d03dfb12" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43069 } }
I20260619 11:59:54.426177 3289 leader_election.cc:302] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af02755a5f334e0ca86d3724d03dfb12; no voters:
I20260619 11:59:54.426389 3289 leader_election.cc:288] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:54.426455 3294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:54.426693 3289 sys_catalog.cc:564] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:54.426703 3294 raft_consensus.cc:686] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [term 1 LEADER]: Becoming Leader. State: Replica: af02755a5f334e0ca86d3724d03dfb12, State: Running, Role: LEADER
I20260619 11:59:54.426949 3294 consensus_queue.cc:237] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af02755a5f334e0ca86d3724d03dfb12" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43069 } }
I20260619 11:59:54.427503 3296 sys_catalog.cc:455] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af02755a5f334e0ca86d3724d03dfb12. Latest consensus state: current_term: 1 leader_uuid: "af02755a5f334e0ca86d3724d03dfb12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af02755a5f334e0ca86d3724d03dfb12" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43069 } } }
I20260619 11:59:54.427618 3296 sys_catalog.cc:458] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:54.427836 3303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:54.428050 3295 sys_catalog.cc:455] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af02755a5f334e0ca86d3724d03dfb12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af02755a5f334e0ca86d3724d03dfb12" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43069 } } }
I20260619 11:59:54.428146 3295 sys_catalog.cc:458] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12 [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:54.428467 3303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:59:54.430591 3303 catalog_manager.cc:1347] Generated new cluster ID: d2a3d00af0a34934a10f8d3da0869048
I20260619 11:59:54.430656 3303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:54.437216 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:54.422972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.0.203.254:43069)
0619 11:59:54.423201 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:54.423211 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:54.423243 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:59:54.424723 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:54.424733 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:54.424798 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:54.424973 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:54.425814 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.427350 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.428517 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.429164 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.429328 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:54.432562 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:54.432567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:54.432567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:54.432583 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:54.435110 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.435608 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.435615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.435615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.435750 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.436006 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.436007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.436007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.436327 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:54.436491 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:54.436705 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 11:59:54.436961 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
I20260619 11:59:54.441901 3303 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:54.442526 3303 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:54.445602 3303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af02755a5f334e0ca86d3724d03dfb12: Generated new TSK 0
I20260619 11:59:54.445848 3303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 11:59:54.476292 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43069
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:54.570514 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:54.570832 3315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:54.570875 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:54.572209 3315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:54.572285 3315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:54.572335 3315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 11:59:54.574036 3315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:54.574337 3315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:54.574599 3315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:54.580986 3323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.581023 3321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.581180 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:54.581382 3315 server_base.cc:1034] running on GCE node
I20260619 11:59:54.581676 3315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:54.581955 3315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:54.583149 3315 hybrid_clock.cc:648] HybridClock initialized: now 1781870394583128 us; error 44 us; skew 500 ppm
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:54.585965 3315 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 11:59:54.586500 3315 webserver.cc:460] Webserver started at http://127.0.203.193:36497/ using document root <none> and password file <none>
I20260619 11:59:54.586705 3315 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:54.586750 3315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:54.586848 3315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:54.587976 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "738cbc7c1cc1481082c9cad435f859e7"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "2e2a830f3e9c0062efe949902b9553ef"
server_key_iv: "fff7c79301e3d4c7dcd1c5c913398469"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.588295 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "738cbc7c1cc1481082c9cad435f859e7"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "2e2a830f3e9c0062efe949902b9553ef"
server_key_iv: "fff7c79301e3d4c7dcd1c5c913398469"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.589929 3315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 11:59:54.590901 3330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.591172 3315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:54.591296 3315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "738cbc7c1cc1481082c9cad435f859e7"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "2e2a830f3e9c0062efe949902b9553ef"
server_key_iv: "fff7c79301e3d4c7dcd1c5c913398469"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.591394 3315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:54.609247 3315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:54.611591 3315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:54.611714 3315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:54.611907 3315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:54.612262 3315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:54.612313 3315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.612337 3315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:54.612393 3315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.619690 3315 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:44311
I20260619 11:59:54.619797 3443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:44311 every 8 connection(s)
I20260619 11:59:54.620173 3315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 11:59:54.622833 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 3315
I20260619 11:59:54.622959 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 11:59:54.623140 815 external_mini_cluster.cc:1428] Setting key 0400a92514b62a48c5c363ba01bf79c5
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 11:59:54.632992 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:54.621162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:54943 (local address 127.0.203.254:43069)
0619 11:59:54.621285 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:54.621288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:54.621497 (+ 209us) server_negotiation.cc:407] Connection header received
0619 11:59:54.622625 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:54.622627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:54.622647 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:54.622697 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:54.623616 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.624291 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.625637 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.625925 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.626023 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:54.629377 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:54.629379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:54.629380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:54.629389 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:54.630951 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.631529 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.631530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.631530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.631595 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.631948 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.631949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.631949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.632142 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:54.632234 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:54.632791 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 11:59:54.632886 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 11:59:54.633157 3446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:54.621081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:43069 (local address 127.0.203.193:54943)
0619 11:59:54.621333 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:54.621381 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:54.622499 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:54.622796 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:54.622800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:54.623171 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:54.623522 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:54.623528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.624402 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.624403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:54.625521 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:54.625524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.626038 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.626039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:54.626201 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:54.626840 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:54.626852 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:54.629230 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:54.631056 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:54.631064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:54.631067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:54.631402 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:54.631688 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:54.631689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:54.631689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:54.631840 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:54.632241 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:54.632248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:54.632467 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 11:59:54.632611 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 11:59:54.632830 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260619 11:59:54.633947 3444 heartbeater.cc:347] Connected to a master server at 127.0.203.254:43069
I20260619 11:59:54.634065 3444 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:54.634294 3444 heartbeater.cc:511] Master 127.0.203.254:43069 requested a full tablet report, sending...
I20260619 11:59:54.635161 3253 ts_manager.cc:194] Registered new tserver with Master: 738cbc7c1cc1481082c9cad435f859e7 (127.0.203.193:44311)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
I20260619 11:59:54.636088 3253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:54943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 11:59:54.652524 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43069
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:54.742851 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:54.743099 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:54.743124 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:54.744302 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:54.744341 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:54.744366 3449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 11:59:54.745862 3449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:54.746074 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:54.746277 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:54.751953 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.751955 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.751932 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:54.752259 3449 server_base.cc:1034] running on GCE node
I20260619 11:59:54.752526 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:54.752861 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:54.754057 3449 hybrid_clock.cc:648] HybridClock initialized: now 1781870394754041 us; error 63 us; skew 500 ppm
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:54.757328 3449 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 11:59:54.757890 3449 webserver.cc:460] Webserver started at http://127.0.203.194:46483/ using document root <none> and password file <none>
I20260619 11:59:54.758145 3449 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:54.758196 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:54.758355 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:54.759541 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "ca77e4d1e0e34806994fb1fef8ca35de"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "09724279e853a56b9a1188b3413ab592"
server_key_iv: "2e7c18234a1bcd6f36d68023b263693f"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.759969 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "ca77e4d1e0e34806994fb1fef8ca35de"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "09724279e853a56b9a1188b3413ab592"
server_key_iv: "2e7c18234a1bcd6f36d68023b263693f"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.761548 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 11:59:54.762440 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.762727 3449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:54.762854 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "ca77e4d1e0e34806994fb1fef8ca35de"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "09724279e853a56b9a1188b3413ab592"
server_key_iv: "2e7c18234a1bcd6f36d68023b263693f"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.762952 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:54.778535 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:54.781144 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:54.781286 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:54.781492 3449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:54.781855 3449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:54.781900 3449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.781925 3449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:54.781985 3449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.789288 3449 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:39459
I20260619 11:59:54.789407 3577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:39459 every 8 connection(s)
I20260619 11:59:54.789760 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:54.798911 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 3449
I20260619 11:59:54.799047 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 11:59:54.799302 815 external_mini_cluster.cc:1428] Setting key 23586853c2798f41b03ba2996b109fb8
I20260619 11:59:54.800568 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:54.790761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:40981 (local address 127.0.203.254:43069)
0619 11:59:54.790868 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:54.790870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:54.791066 (+ 196us) server_negotiation.cc:407] Connection header received
0619 11:59:54.792152 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:54.792153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:54.792166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:54.792264 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:54.793095 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.793696 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.794822 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.795037 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.795140 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:54.797584 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:54.797585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:54.797585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:54.797594 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:54.798946 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.799551 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.799552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.799553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.799611 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.799888 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.799889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.799889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.800007 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:54.800084 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:54.800378 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 11:59:54.800462 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:59:54.800885 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:54.790693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:43069 (local address 127.0.203.194:40981)
0619 11:59:54.790895 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:54.790943 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:54.792011 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:54.792349 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:54.792353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:54.792715 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:54.792965 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:54.792969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.793817 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.793818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:54.794733 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:54.794735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.795156 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.795156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:54.795261 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:54.795871 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:54.795880 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:54.797488 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:54.799051 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:54.799066 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:54.799069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:54.799470 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:54.799684 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:54.799685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:54.799685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:54.799818 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:54.800074 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:54.800080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:54.800244 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 11:59:54.800360 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 11:59:54.800550 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":115,"threads_started":1}
I20260619 11:59:54.801440 3578 heartbeater.cc:347] Connected to a master server at 127.0.203.254:43069
I20260619 11:59:54.801577 3578 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:54.801817 3578 heartbeater.cc:511] Master 127.0.203.254:43069 requested a full tablet report, sending...
I20260619 11:59:54.802341 3253 ts_manager.cc:194] Registered new tserver with Master: ca77e4d1e0e34806994fb1fef8ca35de (127.0.203.194:39459)
I20260619 11:59:54.802932 3253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:40981
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 11:59:54.829077 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43069
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:54.924011 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:54.924273 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:54.924297 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:54.925590 3583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:54.925637 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:54.925664 3583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 11:59:54.927271 3583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:54.927552 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:54.927915 3583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:54.933607 3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.933619 3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:54.933612 3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:54.934772 3583 server_base.cc:1034] running on GCE node
I20260619 11:59:54.935063 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:54.935318 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:54.936525 3583 hybrid_clock.cc:648] HybridClock initialized: now 1781870394936501 us; error 37 us; skew 500 ppm
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:54.939572 3583 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 11:59:54.940097 3583 webserver.cc:460] Webserver started at http://127.0.203.195:39963/ using document root <none> and password file <none>
I20260619 11:59:54.940366 3583 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:54.940440 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:54.940588 3583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:54.941736 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "21c806ec831f4a3cbe65cd65eb080faf"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "d43319251579ad3eaac9916961bfccb7"
server_key_iv: "cc263550cf006b7ffc6280c3a4f4dac7"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.942121 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "21c806ec831f4a3cbe65cd65eb080faf"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "d43319251579ad3eaac9916961bfccb7"
server_key_iv: "cc263550cf006b7ffc6280c3a4f4dac7"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.943675 3583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 11:59:54.944554 3598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.944852 3583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:54.944967 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "21c806ec831f4a3cbe65cd65eb080faf"
format_stamp: "Formatted at 2026-06-19 11:59:54 on dist-test-slave-fbq8"
server_key: "d43319251579ad3eaac9916961bfccb7"
server_key_iv: "cc263550cf006b7ffc6280c3a4f4dac7"
server_key_version: "encryptionkey@0"
I20260619 11:59:54.945071 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:54.959113 3583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:54.961498 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:54.961704 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:54.961979 3583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:54.962347 3583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:54.962384 3583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.962410 3583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:54.962457 3583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:54.969731 3583 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:45461
I20260619 11:59:54.969900 3711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:45461 every 8 connection(s)
I20260619 11:59:54.970187 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 11:59:54.975356 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 3583
I20260619 11:59:54.975454 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 11:59:54.975667 815 external_mini_cluster.cc:1428] Setting key fe19330f3f53871480e3bb434b95e69d
Jun 19 11:59:54 dist-test-slave-fbq8 krb5kdc[3208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:54.980916 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:54.971044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:37939 (local address 127.0.203.254:43069)
0619 11:59:54.971108 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:54.971110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:54.971359 (+ 249us) server_negotiation.cc:407] Connection header received
0619 11:59:54.972407 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:54.972408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:54.972420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:54.972496 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:54.973278 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.973817 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.974889 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.975135 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.975217 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:54.977997 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:54.977999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:54.977999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:54.978007 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:54.979411 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.979801 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.979802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.979803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.979865 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:54.980162 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:54.980163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:54.980163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:54.980304 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:54.980383 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:54.980717 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 11:59:54.980843 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:59:54.980969 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:54.971007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:43069 (local address 127.0.203.195:37939)
0619 11:59:54.971199 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:54.971240 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:54.972263 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:54.972562 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:54.972566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:54.972939 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:54.973162 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:54.973166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.973907 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.973908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:54.974798 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:54.974801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:54.975231 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:54.975231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:54.975334 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:54.976115 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:54.976125 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:54.977893 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:54.979495 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:54.979500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:54.979502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:54.979720 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:54.979933 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:54.979934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:54.979934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:54.980027 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:54.980380 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:54.980384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:54.980505 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 11:59:54.980600 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 11:59:54.980790 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":120,"threads_started":1}
I20260619 11:59:54.981535 3712 heartbeater.cc:347] Connected to a master server at 127.0.203.254:43069
I20260619 11:59:54.981696 3712 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:54.981953 3712 heartbeater.cc:511] Master 127.0.203.254:43069 requested a full tablet report, sending...
I20260619 11:59:54.982440 3253 ts_manager.cc:194] Registered new tserver with Master: 21c806ec831f4a3cbe65cd65eb080faf (127.0.203.195:45461)
I20260619 11:59:54.982879 3253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:37939
I20260619 11:59:54.988019 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:59:55.001922 3293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:59:55.000388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.0.203.254:43069)
0619 11:59:55.000515 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:55.000518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:55.000631 (+ 113us) server_negotiation.cc:407] Connection header received
0619 11:59:55.000757 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:55.000758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:55.000769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:55.000826 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:55.001846 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46870: BlockingRecv error: recv got EOF from 127.0.0.1:46870 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 11:59:55.003283 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 3315
I20260619 11:59:55.009881 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 3449
I20260619 11:59:55.015769 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 3583
I20260619 11:59:55.021328 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 3221
2026-06-19T11:59:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1951 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:55 dist-test-slave-fbq8 krb5kdc[3724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:55 dist-test-slave-fbq8 krb5kdc[3724](info): set up 2 sockets
Jun 19 11:59:55 dist-test-slave-fbq8 krb5kdc[3724](info): commencing operation
krb5kdc: starting...
W20260619 11:59:56.128723 815 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:56 dist-test-slave-fbq8 krb5kdc[3724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:56.260316 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42573
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:42573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:56.346904 3737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:56.347168 3737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:56.347208 3737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:56.348431 3737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:56.348502 3737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:56.348531 3737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:56.348562 3737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:56.348590 3737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:56.350186 3737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:42573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:56.350482 3737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:56.350733 3737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:56.356314 3742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:56.356302 3743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:56.356420 3737 server_base.cc:1034] running on GCE node
W20260619 11:59:56.356307 3745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:56.357080 3737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:56.357427 3737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:56.358609 3737 hybrid_clock.cc:648] HybridClock initialized: now 1781870396358564 us; error 53 us; skew 500 ppm
Jun 19 11:59:56 dist-test-slave-fbq8 krb5kdc[3724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:56.369170 3737 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:56.369767 3737 webserver.cc:460] Webserver started at http://127.0.203.254:34935/ using document root <none> and password file <none>
I20260619 11:59:56.370040 3737 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:56.370114 3737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:56.370280 3737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:56.371544 3737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6"
format_stamp: "Formatted at 2026-06-19 11:59:56 on dist-test-slave-fbq8"
server_key: "311134d40a5a671b86e914c70b33f3df"
server_key_iv: "1d6803cbb1d37206d15ca24c1898ae64"
server_key_version: "encryptionkey@0"
I20260619 11:59:56.371943 3737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6"
format_stamp: "Formatted at 2026-06-19 11:59:56 on dist-test-slave-fbq8"
server_key: "311134d40a5a671b86e914c70b33f3df"
server_key_iv: "1d6803cbb1d37206d15ca24c1898ae64"
server_key_version: "encryptionkey@0"
I20260619 11:59:56.373682 3737 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:59:56.374686 3752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:56.375028 3737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:56.375167 3737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6"
format_stamp: "Formatted at 2026-06-19 11:59:56 on dist-test-slave-fbq8"
server_key: "311134d40a5a671b86e914c70b33f3df"
server_key_iv: "1d6803cbb1d37206d15ca24c1898ae64"
server_key_version: "encryptionkey@0"
I20260619 11:59:56.375252 3737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:56.400501 3737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:56.403378 3737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:56.403577 3737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:56.409169 3737 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:42573
I20260619 11:59:56.409250 3804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:42573 every 8 connection(s)
I20260619 11:59:56.409686 3737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:56.410743 3805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:56.413522 3805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6: Bootstrap starting.
I20260619 11:59:56.414201 3805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:56.414517 3805 log.cc:824] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:56.415446 3805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6: No bootstrap required, opened a new log
I20260619 11:59:56.416613 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 3737
I20260619 11:59:56.416782 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:56.417007 815 external_mini_cluster.cc:1428] Setting key 1b3b1efe20704d31acc33eed2119d9f5
I20260619 11:59:56.417537 3805 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42573 } }
I20260619 11:59:56.417807 3805 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:56.417858 3805 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5eb2cccbd14f8f8e3cddc037fd26a6, State: Initialized, Role: FOLLOWER
I20260619 11:59:56.418020 3805 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42573 } }
I20260619 11:59:56.418128 3805 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:56.418174 3805 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:56.418226 3805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:56.418936 3805 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42573 } }
I20260619 11:59:56.419076 3805 leader_election.cc:302] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea5eb2cccbd14f8f8e3cddc037fd26a6; no voters:
I20260619 11:59:56.419322 3805 leader_election.cc:288] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:56.419394 3810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:56.419667 3805 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:56.419835 3810 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [term 1 LEADER]: Becoming Leader. State: Replica: ea5eb2cccbd14f8f8e3cddc037fd26a6, State: Running, Role: LEADER
I20260619 11:59:56.419965 3810 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42573 } }
I20260619 11:59:56.420553 3810 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5eb2cccbd14f8f8e3cddc037fd26a6. Latest consensus state: current_term: 1 leader_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42573 } } }
I20260619 11:59:56.420791 3810 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:56.420601 3811 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5eb2cccbd14f8f8e3cddc037fd26a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42573 } } }
I20260619 11:59:56.420974 3811 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6 [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:56.421095 3819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:56.421679 3819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:59:56.423681 3819 catalog_manager.cc:1347] Generated new cluster ID: a7565453beeb4b559aec0ee16339645c
I20260619 11:59:56.423748 3819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 11:59:56 dist-test-slave-fbq8 krb5kdc[3724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:56.430236 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:56.417735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.0.203.254:42573)
0619 11:59:56.418420 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:56.418430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:56.418463 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:59:56.418604 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:56.418611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:56.418665 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:56.418833 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:56.419738 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:56.421382 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:56.422503 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:56.423137 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:56.423328 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:56.426474 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:56.426487 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:56.426487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:56.426501 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:56.428416 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:56.428929 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:56.428932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:56.428932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:56.429051 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:56.429265 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:56.429266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:56.429266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:56.429557 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:56.429763 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:56.429837 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 11:59:56.430018 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":586,"threads_started":1}
I20260619 11:59:56.431509 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:59:56.449724 3819 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:56.450286 3819 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:56.452316 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:56.447865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.0.203.254:42573)
0619 11:59:56.447928 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:56.447931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:56.447940 (+ 9us) server_negotiation.cc:407] Connection header received
0619 11:59:56.447969 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:56.447970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:56.447994 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:56.448050 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:56.448673 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:56.449292 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:56.450482 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:56.450814 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:56.450879 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:56.451762 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:56.451764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:56.451764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:59:56.452008 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:56.452030 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:59:56.452037 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:56.452099 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:56.452187 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 11:59:56.452237 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 11:59:56.460633 3819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea5eb2cccbd14f8f8e3cddc037fd26a6: Generated new TSK 0
I20260619 11:59:56.460888 3819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:59:56.469810 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 3737
2026-06-19T11:59:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1444 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:56 dist-test-slave-fbq8 krb5kdc[3838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:56 dist-test-slave-fbq8 krb5kdc[3838](info): set up 2 sockets
Jun 19 11:59:56 dist-test-slave-fbq8 krb5kdc[3838](info): commencing operation
krb5kdc: starting...
W20260619 11:59:57.584581 815 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:57 dist-test-slave-fbq8 krb5kdc[3838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:57.715492 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44597
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:57.805961 3851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:57.806296 3851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:57.806349 3851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:57.808316 3851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:57.808409 3851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:57.808437 3851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:57.808468 3851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:57.808495 3851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:57.811913 3851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:57.812291 3851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:57.812587 3851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:57.818369 3857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:57.818346 3859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:57.818378 3856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:57.818686 3851 server_base.cc:1034] running on GCE node
I20260619 11:59:57.819031 3851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:57.819332 3851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:57.820495 3851 hybrid_clock.cc:648] HybridClock initialized: now 1781870397820460 us; error 58 us; skew 500 ppm
Jun 19 11:59:57 dist-test-slave-fbq8 krb5kdc[3838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:57.823518 3851 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:57.824165 3851 webserver.cc:460] Webserver started at http://127.0.203.254:38391/ using document root <none> and password file <none>
I20260619 11:59:57.824473 3851 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:57.824538 3851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:57.824754 3851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:57.825925 3851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "dc5a5128f59843e783b62942d4005935"
format_stamp: "Formatted at 2026-06-19 11:59:57 on dist-test-slave-fbq8"
server_key: "cdec9e8d8c0796799f5451a327c47043"
server_key_iv: "ad5fe9335c66e10fde9fbe1bd6c0eefe"
server_key_version: "encryptionkey@0"
I20260619 11:59:57.826522 3851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "dc5a5128f59843e783b62942d4005935"
format_stamp: "Formatted at 2026-06-19 11:59:57 on dist-test-slave-fbq8"
server_key: "cdec9e8d8c0796799f5451a327c47043"
server_key_iv: "ad5fe9335c66e10fde9fbe1bd6c0eefe"
server_key_version: "encryptionkey@0"
I20260619 11:59:57.828163 3851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 11:59:57.829277 3866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:57.829630 3851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 11:59:57.829758 3851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "dc5a5128f59843e783b62942d4005935"
format_stamp: "Formatted at 2026-06-19 11:59:57 on dist-test-slave-fbq8"
server_key: "cdec9e8d8c0796799f5451a327c47043"
server_key_iv: "ad5fe9335c66e10fde9fbe1bd6c0eefe"
server_key_version: "encryptionkey@0"
I20260619 11:59:57.829844 3851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:57.849243 3851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:57.851435 3851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:57.851616 3851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:57.856372 3851 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:44597
I20260619 11:59:57.856437 3918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:44597 every 8 connection(s)
I20260619 11:59:57.856916 3851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:57.858161 3919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:57.861028 3919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935: Bootstrap starting.
I20260619 11:59:57.861708 3919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:57.862035 3919 log.cc:824] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:57.863070 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 3851
I20260619 11:59:57.863235 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:57.863377 3919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935: No bootstrap required, opened a new log
I20260619 11:59:57.863494 815 external_mini_cluster.cc:1428] Setting key e7c6b4a7a62dbc53b57e7b890dee5a69
I20260619 11:59:57.865962 3919 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5a5128f59843e783b62942d4005935" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44597 } }
I20260619 11:59:57.866173 3919 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:57.866222 3919 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5a5128f59843e783b62942d4005935, State: Initialized, Role: FOLLOWER
I20260619 11:59:57.866425 3919 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5a5128f59843e783b62942d4005935" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44597 } }
I20260619 11:59:57.866530 3919 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:57.866573 3919 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:57.866626 3919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:57.867372 3919 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5a5128f59843e783b62942d4005935" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44597 } }
I20260619 11:59:57.867506 3919 leader_election.cc:302] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc5a5128f59843e783b62942d4005935; no voters:
I20260619 11:59:57.867753 3919 leader_election.cc:288] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:57.867784 3924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:57.867916 3924 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [term 1 LEADER]: Becoming Leader. State: Replica: dc5a5128f59843e783b62942d4005935, State: Running, Role: LEADER
I20260619 11:59:57.868073 3924 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5a5128f59843e783b62942d4005935" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44597 } }
I20260619 11:59:57.868078 3919 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:57.869642 3926 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc5a5128f59843e783b62942d4005935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5a5128f59843e783b62942d4005935" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44597 } } }
I20260619 11:59:57.869788 3926 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:57.870055 3934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:57.870734 3934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:59:57.869653 3925 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5a5128f59843e783b62942d4005935. Latest consensus state: current_term: 1 leader_uuid: "dc5a5128f59843e783b62942d4005935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5a5128f59843e783b62942d4005935" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44597 } } }
I20260619 11:59:57.871412 3925 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935 [sys.catalog]: This master's current role is: LEADER
Jun 19 11:59:57 dist-test-slave-fbq8 krb5kdc[3838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:57.872769 3934 catalog_manager.cc:1347] Generated new cluster ID: 2a5d0057a2cb40af89ef1716ad7b8136
I20260619 11:59:57.872848 3934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:59:57.877197 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:57.864093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.0.203.254:44597)
0619 11:59:57.864292 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:57.864303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:57.864381 (+ 78us) server_negotiation.cc:407] Connection header received
0619 11:59:57.865079 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:57.865087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:57.865153 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:57.865320 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:57.865969 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:57.867372 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:57.868392 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:57.869063 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:57.869179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:57.872716 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:57.872721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:57.872721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:57.872737 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:57.874951 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:57.875637 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:57.875641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:57.875641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:57.875778 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:57.876023 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:57.876024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:57.876024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:57.876388 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:57.876590 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:57.876763 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 11:59:57.876942 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260619 11:59:57.878563 3934 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:57.879336 3934 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:57.886096 3934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc5a5128f59843e783b62942d4005935: Generated new TSK 0
I20260619 11:59:57.886314 3934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 11:59:57.921345 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44597
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:58.014424 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:58.014640 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:58.014663 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:58.015815 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:58.015862 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:58.015889 3945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 11:59:58.017552 3945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:58.017817 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:58.018075 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:58.023370 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:58.023372 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:58.023633 3945 server_base.cc:1034] running on GCE node
W20260619 11:59:58.023372 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:58.023984 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:58.024313 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:58.025513 3945 hybrid_clock.cc:648] HybridClock initialized: now 1781870398025476 us; error 62 us; skew 500 ppm
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:58.028611 3945 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 11:59:58.029443 3945 webserver.cc:460] Webserver started at http://127.0.203.193:44533/ using document root <none> and password file <none>
I20260619 11:59:58.029677 3945 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:58.029722 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:58.029958 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:58.031214 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "4e73bbb34611474b9bf38b6911d9791c"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "d5756236630f2639b7bec719ef37408b"
server_key_iv: "0b532b773744f61fbfd508ef23b9a8c0"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.031567 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "4e73bbb34611474b9bf38b6911d9791c"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "d5756236630f2639b7bec719ef37408b"
server_key_iv: "0b532b773744f61fbfd508ef23b9a8c0"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.033249 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:59:58.034189 3960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.034453 3945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:58.034538 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "4e73bbb34611474b9bf38b6911d9791c"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "d5756236630f2639b7bec719ef37408b"
server_key_iv: "0b532b773744f61fbfd508ef23b9a8c0"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.034662 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:58.047797 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:58.050537 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:58.050668 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:58.050954 3945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:58.051306 3945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:58.051363 3945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.051401 3945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:58.051450 3945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.058333 3945 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:46835
I20260619 11:59:58.058462 4073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:46835 every 8 connection(s)
I20260619 11:59:58.058784 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:58.068943 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 3945
I20260619 11:59:58.069044 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 11:59:58.069264 815 external_mini_cluster.cc:1428] Setting key ff5f481c49250c139d94ed33c51d6aa1
I20260619 11:59:58.070029 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:58.059669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:44213 (local address 127.0.203.254:44597)
0619 11:59:58.059778 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:58.059781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:58.059997 (+ 216us) server_negotiation.cc:407] Connection header received
0619 11:59:58.060609 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:58.060611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:58.060622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:58.060765 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:58.061944 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.062479 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.063621 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.063860 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.063925 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:58.066382 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:58.066384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:58.066384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:58.066392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:58.068016 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.068691 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.068693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.068693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.068771 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.069243 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.069245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.069245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.069395 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:58.069490 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:58.069814 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 11:59:58.069925 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 11:59:58.070252 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:58.059620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44597 (local address 127.0.203.193:44213)
0619 11:59:58.059834 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:58.059878 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:58.060474 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:58.060878 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:58.060882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:58.061602 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:58.061835 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:58.061840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.062593 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.062594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:58.063523 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:58.063526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.063946 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.063946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:58.064055 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:58.064696 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:58.064708 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:58.066258 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:58.068118 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:58.068125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:58.068127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:58.068531 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:58.068915 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:58.068916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:58.068917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:58.069165 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:58.069480 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:58.069488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:58.069666 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 11:59:58.069783 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 11:59:58.069996 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":116,"threads_started":1}
I20260619 11:59:58.070864 4074 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44597
I20260619 11:59:58.071000 4074 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:58.071195 4074 heartbeater.cc:511] Master 127.0.203.254:44597 requested a full tablet report, sending...
I20260619 11:59:58.071997 3883 ts_manager.cc:194] Registered new tserver with Master: 4e73bbb34611474b9bf38b6911d9791c (127.0.203.193:46835)
I20260619 11:59:58.073213 3883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:44213
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 11:59:58.097693 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44597
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:58.188369 4079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:58.188627 4079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:58.188728 4079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:58.189888 4079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:58.189929 4079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:58.189954 4079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 11:59:58.191385 4079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:58.191605 4079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:58.191812 4079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:58.197461 4084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:58.197466 4085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:58.197471 4087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:58.197511 4079 server_base.cc:1034] running on GCE node
I20260619 11:59:58.197981 4079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:58.198254 4079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:58.199394 4079 hybrid_clock.cc:648] HybridClock initialized: now 1781870398199377 us; error 37 us; skew 500 ppm
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:58.202226 4079 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 11:59:58.202749 4079 webserver.cc:460] Webserver started at http://127.0.203.194:34017/ using document root <none> and password file <none>
I20260619 11:59:58.202996 4079 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:58.203070 4079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:58.203229 4079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:58.204366 4079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "6b48ebe1723e4e3f82403457f748df16"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "02ad7d9945c5a0f48f550ee587ecea9d"
server_key_iv: "c2bda4b414d11491acf820f910424f6a"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.204803 4079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "6b48ebe1723e4e3f82403457f748df16"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "02ad7d9945c5a0f48f550ee587ecea9d"
server_key_iv: "c2bda4b414d11491acf820f910424f6a"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.206277 4079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:58.207101 4094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.207355 4079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:58.207458 4079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "6b48ebe1723e4e3f82403457f748df16"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "02ad7d9945c5a0f48f550ee587ecea9d"
server_key_iv: "c2bda4b414d11491acf820f910424f6a"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.207551 4079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:58.228107 4079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:58.231137 4079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:58.231338 4079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:58.231633 4079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:58.232004 4079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:58.232064 4079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.232120 4079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:58.232151 4079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.240949 4079 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:36467
I20260619 11:59:58.240998 4207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:36467 every 8 connection(s)
I20260619 11:59:58.241371 4079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 11:59:58.244248 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4079
I20260619 11:59:58.244381 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 11:59:58.244593 815 external_mini_cluster.cc:1428] Setting key 288757b36fef8adea57f24cfadc6c0b7
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:58.254318 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:58.242273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:34025 (local address 127.0.203.254:44597)
0619 11:59:58.242372 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:58.242375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:58.242624 (+ 249us) server_negotiation.cc:407] Connection header received
0619 11:59:58.243180 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:58.243181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:58.243193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:58.243261 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:58.244440 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.245294 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.246886 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.247251 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.247338 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:58.250660 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:58.250662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:58.250663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:58.250675 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:58.252612 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.253125 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.253126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.253126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.253195 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.253465 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.253466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.253466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.253634 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:58.253721 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:58.254091 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 11:59:58.254183 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:59:58.254468 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:58.242241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44597 (local address 127.0.203.194:34025)
0619 11:59:58.242459 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:58.242501 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:58.243043 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:58.243324 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:58.243328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:58.243951 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:58.244296 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:58.244303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.245417 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.245419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:58.246776 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:58.246780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.247353 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.247354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:58.247471 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:58.248290 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:58.248304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:58.250530 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:58.252771 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:58.252776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:58.252779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:58.253025 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:58.253278 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:58.253279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:58.253279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:58.253382 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:58.253755 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:58.253761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:58.253923 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 11:59:58.254044 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 11:59:58.254201 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":116,"threads_started":1}
I20260619 11:59:58.255236 4208 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44597
I20260619 11:59:58.255340 4208 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:58.255553 4208 heartbeater.cc:511] Master 127.0.203.254:44597 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
I20260619 11:59:58.256043 3883 ts_manager.cc:194] Registered new tserver with Master: 6b48ebe1723e4e3f82403457f748df16 (127.0.203.194:36467)
I20260619 11:59:58.256444 3883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:34025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 11:59:58.277604 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44597
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:58.365738 4213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:58.365965 4213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:58.365993 4213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:58.367187 4213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:58.367234 4213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:58.367264 4213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 11:59:58.369398 4213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:58.369691 4213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:58.369995 4213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:58.375267 4221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:58.375259 4219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:58.375267 4218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:58.375587 4213 server_base.cc:1034] running on GCE node
I20260619 11:59:58.375873 4213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:58.376135 4213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:58.377317 4213 hybrid_clock.cc:648] HybridClock initialized: now 1781870398377302 us; error 32 us; skew 500 ppm
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:58.380080 4213 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 11:59:58.380550 4213 webserver.cc:460] Webserver started at http://127.0.203.195:33085/ using document root <none> and password file <none>
I20260619 11:59:58.380839 4213 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:58.380916 4213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:58.381067 4213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:58.382146 4213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "2e931464189c44bbb092c57bb5867716"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "1deae74eab5dfe7dc02008f7e887e413"
server_key_iv: "64bf49a998229b1b90e7900642900d67"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.382526 4213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "2e931464189c44bbb092c57bb5867716"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "1deae74eab5dfe7dc02008f7e887e413"
server_key_iv: "64bf49a998229b1b90e7900642900d67"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.383900 4213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 11:59:58.384760 4228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.384960 4213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:58.385068 4213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "2e931464189c44bbb092c57bb5867716"
format_stamp: "Formatted at 2026-06-19 11:59:58 on dist-test-slave-fbq8"
server_key: "1deae74eab5dfe7dc02008f7e887e413"
server_key_iv: "64bf49a998229b1b90e7900642900d67"
server_key_version: "encryptionkey@0"
I20260619 11:59:58.385161 4213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:58.392879 4213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:58.395216 4213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:58.395387 4213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:58.395624 4213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:59:58.395955 4213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:59:58.396010 4213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.396052 4213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:59:58.396080 4213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:58.402796 4213 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:36481
I20260619 11:59:58.402854 4341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:36481 every 8 connection(s)
I20260619 11:59:58.403218 4213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:58.412902 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4213
I20260619 11:59:58.413010 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 11:59:58.413239 815 external_mini_cluster.cc:1428] Setting key 37c0cd648177d457ea0a22ddc2adce39
I20260619 11:59:58.413950 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:58.404144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:54033 (local address 127.0.203.254:44597)
0619 11:59:58.404226 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:58.404229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:58.404435 (+ 206us) server_negotiation.cc:407] Connection header received
0619 11:59:58.405074 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:58.405076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:58.405087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:58.405156 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:58.406111 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.406691 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.407735 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.407959 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.408021 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:58.410527 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:58.410528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:58.410529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:58.410535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:58.412119 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.412559 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.412560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.412561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.412617 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.413074 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.413075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.413076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.413226 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:58.413291 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:58.413719 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 11:59:58.413841 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:59:58.414103 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:58.404119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44597 (local address 127.0.203.195:54033)
0619 11:59:58.404306 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 11:59:58.404344 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:59:58.404961 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:59:58.405210 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:59:58.405213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:59:58.405809 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:59:58.406009 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:58.406013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.406811 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.406812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:58.407644 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:59:58.407646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.408040 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.408041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:59:58.408143 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:58.408784 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:59:58.408792 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:59:58.410290 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:59:58.412226 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:58.412233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:58.412235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:58.412480 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:58.412798 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:59:58.412799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:59:58.412800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:59:58.412961 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:59:58.413301 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:59:58.413308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:59:58.413512 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 11:59:58.413635 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 11:59:58.413832 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":114,"threads_started":1}
I20260619 11:59:58.414572 4342 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44597
I20260619 11:59:58.414709 4342 heartbeater.cc:464] Registering TS with master...
I20260619 11:59:58.414896 4342 heartbeater.cc:511] Master 127.0.203.254:44597 requested a full tablet report, sending...
I20260619 11:59:58.415304 3883 ts_manager.cc:194] Registered new tserver with Master: 2e931464189c44bbb092c57bb5867716 (127.0.203.195:36481)
I20260619 11:59:58.415689 3883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:54033
I20260619 11:59:58.425585 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[3838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870398, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:58.449414 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:58.441857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.203.254:44597)
0619 11:59:58.441922 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:58.441924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:58.442055 (+ 131us) server_negotiation.cc:407] Connection header received
0619 11:59:58.442088 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:58.442089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:58.442100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:58.442155 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:58.442786 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.443290 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.444114 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:58.444301 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:58.444371 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:58.446794 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:58.446796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:58.446796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:58.446803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:58.448174 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.448493 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.448494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.448494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.448545 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:58.448770 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:58.448771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:58.448771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:58.448954 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:58.449035 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:58.449273 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 11:59:58.449344 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 11:59:58.449715 3883 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48958
I20260619 11:59:58.450314 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 3945
I20260619 11:59:58.455343 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4079
I20260619 11:59:58.459627 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4213
I20260619 11:59:58.463934 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 3851
2026-06-19T11:59:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1996 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[4354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[4354](info): set up 2 sockets
Jun 19 11:59:58 dist-test-slave-fbq8 krb5kdc[4354](info): commencing operation
krb5kdc: starting...
W20260619 11:59:59.524855 815 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 11:59:59 dist-test-slave-fbq8 krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:59:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 11:59:59.660348 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40251
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:59.747726 4367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:59.748035 4367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:59.748078 4367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:59.749423 4367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:59:59.749516 4367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:59.749559 4367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:59.749608 4367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:59:59.749634 4367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:59:59.751112 4367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:40251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:59.751406 4367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:59.751647 4367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:59.756925 4373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:59.756886 4372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:59.757165 4367 server_base.cc:1034] running on GCE node
W20260619 11:59:59.756889 4375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:59.757568 4367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:59.757900 4367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:59.759059 4367 hybrid_clock.cc:648] HybridClock initialized: now 1781870399759003 us; error 84 us; skew 500 ppm
Jun 19 11:59:59 dist-test-slave-fbq8 krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:59.762073 4367 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 11:59:59.762585 4367 webserver.cc:460] Webserver started at http://127.0.203.254:34195/ using document root <none> and password file <none>
I20260619 11:59:59.762802 4367 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:59.762849 4367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:59.762950 4367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:59.764060 4367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "5af13fd78fcb44b183cea82e740d9eca"
format_stamp: "Formatted at 2026-06-19 11:59:59 on dist-test-slave-fbq8"
server_key: "4136b45ba25bd3ebc23900432cb63428"
server_key_iv: "712e8e6643ff336e652aa89b44a962d3"
server_key_version: "encryptionkey@0"
I20260619 11:59:59.764407 4367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "5af13fd78fcb44b183cea82e740d9eca"
format_stamp: "Formatted at 2026-06-19 11:59:59 on dist-test-slave-fbq8"
server_key: "4136b45ba25bd3ebc23900432cb63428"
server_key_iv: "712e8e6643ff336e652aa89b44a962d3"
server_key_version: "encryptionkey@0"
I20260619 11:59:59.766006 4367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 11:59:59.767050 4382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:59.767480 4367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:59:59.767611 4367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "5af13fd78fcb44b183cea82e740d9eca"
format_stamp: "Formatted at 2026-06-19 11:59:59 on dist-test-slave-fbq8"
server_key: "4136b45ba25bd3ebc23900432cb63428"
server_key_iv: "712e8e6643ff336e652aa89b44a962d3"
server_key_version: "encryptionkey@0"
I20260619 11:59:59.767724 4367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:59:59.781517 4367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:59:59.783900 4367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:59:59.784104 4367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:59:59.789240 4367 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:40251
I20260619 11:59:59.789295 4434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:40251 every 8 connection(s)
I20260619 11:59:59.789707 4367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 11:59:59.790782 4435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:59:59.793675 4435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca: Bootstrap starting.
I20260619 11:59:59.794390 4435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca: Neither blocks nor log segments found. Creating new log.
I20260619 11:59:59.794729 4435 log.cc:824] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca: Log is configured to *not* fsync() on all Append() calls
I20260619 11:59:59.795661 4435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca: No bootstrap required, opened a new log
I20260619 11:59:59.796507 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4367
I20260619 11:59:59.796667 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 11:59:59.796916 815 external_mini_cluster.cc:1428] Setting key 6b1c9e718871f9c1e8132a69069c1e02
I20260619 11:59:59.798238 4435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af13fd78fcb44b183cea82e740d9eca" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40251 } }
I20260619 11:59:59.798421 4435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:59:59.798470 4435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af13fd78fcb44b183cea82e740d9eca, State: Initialized, Role: FOLLOWER
I20260619 11:59:59.798681 4435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af13fd78fcb44b183cea82e740d9eca" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40251 } }
I20260619 11:59:59.798784 4435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:59:59.798818 4435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:59:59.798864 4435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:59:59.799973 4435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af13fd78fcb44b183cea82e740d9eca" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40251 } }
I20260619 11:59:59.800084 4435 leader_election.cc:302] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af13fd78fcb44b183cea82e740d9eca; no voters:
I20260619 11:59:59.800282 4435 leader_election.cc:288] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:59:59.800369 4440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:59:59.800586 4435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:59:59.800707 4440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [term 1 LEADER]: Becoming Leader. State: Replica: 5af13fd78fcb44b183cea82e740d9eca, State: Running, Role: LEADER
I20260619 11:59:59.800977 4440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af13fd78fcb44b183cea82e740d9eca" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40251 } }
I20260619 11:59:59.801695 4441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af13fd78fcb44b183cea82e740d9eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af13fd78fcb44b183cea82e740d9eca" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40251 } } }
I20260619 11:59:59.801805 4441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:59.802263 4442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af13fd78fcb44b183cea82e740d9eca. Latest consensus state: current_term: 1 leader_uuid: "5af13fd78fcb44b183cea82e740d9eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af13fd78fcb44b183cea82e740d9eca" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40251 } } }
I20260619 11:59:59.802559 4442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca [sys.catalog]: This master's current role is: LEADER
I20260619 11:59:59.802605 4451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:59:59.803058 4451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:59:59.805111 4451 catalog_manager.cc:1347] Generated new cluster ID: 7a7abcc475ef46dfa1a63d690c31912e
I20260619 11:59:59.805166 4451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 11:59:59 dist-test-slave-fbq8 krb5kdc[4354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 11:59:59.810492 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:59:59.797647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.0.203.254:40251)
0619 11:59:59.797906 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 11:59:59.797915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:59:59.797950 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:59:59.798220 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:59:59.798227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:59:59.798284 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:59:59.798453 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:59:59.799324 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:59.800603 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:59.801891 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:59:59.802618 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:59:59.802751 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:59:59.806039 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:59:59.806043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:59:59.806044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:59:59.806058 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:59:59.808315 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:59.808831 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:59.808835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:59.808835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:59.808991 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:59:59.809303 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:59:59.809304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:59:59.809304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:59:59.809734 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:59:59.809923 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 11:59:59.810096 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 11:59:59.810262 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
I20260619 11:59:59.817062 4451 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:59:59.817960 4451 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:59:59.822615 4451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af13fd78fcb44b183cea82e740d9eca: Generated new TSK 0
I20260619 11:59:59.822824 4451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 11:59:59.861290 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40251
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:59:59.951328 4462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:59:59.951550 4462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:59:59.951573 4462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:59:59.952666 4462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:59:59.952701 4462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:59:59.952726 4462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 11:59:59.954392 4462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 11:59:59.954602 4462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:59:59.954794 4462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:59:59.960196 4467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:59.960203 4468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:59:59.960202 4470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:59:59.960582 4462 server_base.cc:1034] running on GCE node
I20260619 11:59:59.960894 4462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:59:59.961200 4462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:59:59.962387 4462 hybrid_clock.cc:648] HybridClock initialized: now 1781870399962359 us; error 54 us; skew 500 ppm
Jun 19 11:59:59 dist-test-slave-fbq8 krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:59:59.965502 4462 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 11:59:59.966073 4462 webserver.cc:460] Webserver started at http://127.0.203.193:40005/ using document root <none> and password file <none>
I20260619 11:59:59.966388 4462 fs_manager.cc:359] Metadata directory not provided
I20260619 11:59:59.966466 4462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:59:59.966646 4462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:59:59.967795 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "fabc48447b03496289f4a52f8c0facc0"
format_stamp: "Formatted at 2026-06-19 11:59:59 on dist-test-slave-fbq8"
server_key: "69c8aa8f79eaaf9c74f7db60eff0eae8"
server_key_iv: "758437cc250495a05f09c1bd74c4e878"
server_key_version: "encryptionkey@0"
I20260619 11:59:59.968165 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "fabc48447b03496289f4a52f8c0facc0"
format_stamp: "Formatted at 2026-06-19 11:59:59 on dist-test-slave-fbq8"
server_key: "69c8aa8f79eaaf9c74f7db60eff0eae8"
server_key_iv: "758437cc250495a05f09c1bd74c4e878"
server_key_version: "encryptionkey@0"
I20260619 11:59:59.969784 4462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 11:59:59.970801 4477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:59:59.971102 4462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 11:59:59.971238 4462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "fabc48447b03496289f4a52f8c0facc0"
format_stamp: "Formatted at 2026-06-19 11:59:59 on dist-test-slave-fbq8"
server_key: "69c8aa8f79eaaf9c74f7db60eff0eae8"
server_key_iv: "758437cc250495a05f09c1bd74c4e878"
server_key_version: "encryptionkey@0"
I20260619 11:59:59.971324 4462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:00.010584 4462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:00.014982 4462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:00.015219 4462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:00.015784 4462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:00.016423 4462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:00.016520 4462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:00.016595 4462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:00.016768 4462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 12:00:00.027210 4462 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:40249
I20260619 12:00:00.027408 4590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:40249 every 8 connection(s)
I20260619 12:00:00.027801 4462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:00.032510 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4462
I20260619 12:00:00.032714 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:00.032939 815 external_mini_cluster.cc:1428] Setting key 43e280a553c085b65eddf14ac5dac0c2
Jun 19 12:00:00 dist-test-slave-fbq8 krb5kdc[4354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:00.045210 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:00.029487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:36781 (local address 127.0.203.254:40251)
0619 12:00:00.029633 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:00.029636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:00.029666 (+ 30us) server_negotiation.cc:407] Connection header received
0619 12:00:00.031028 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:00.031030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:00.031053 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:00.031151 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:00.032304 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.033180 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.035234 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.035617 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.035728 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:00.039774 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:00.039777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:00.039777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:00.039795 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:00.042924 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:00.043535 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:00.043536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:00.043537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:00.043610 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:00.044005 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:00.044007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:00.044008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:00.044228 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:00.044355 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:00.044871 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 12:00:00.045045 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 12:00:00.045240 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:00.029341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40251 (local address 127.0.203.193:36781)
0619 12:00:00.029525 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:00.029571 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:00.030747 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:00.031452 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:00.031457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:00.031858 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:00.032078 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:00.032084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.033468 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.033525 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:00.035006 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:00.035011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.035873 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.035875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:00.036013 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:00.037182 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:00.037192 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:00.039601 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:00.043100 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:00.043114 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:00.043116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:00.043434 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:00.043749 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:00.043750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:00.043750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:00.043877 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:00.044390 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:00.044397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:00.044563 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 12:00:00.044820 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 12:00:00.045018 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 12:00:00.046525 4591 heartbeater.cc:347] Connected to a master server at 127.0.203.254:40251
I20260619 12:00:00.046705 4591 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:00.046989 4591 heartbeater.cc:511] Master 127.0.203.254:40251 requested a full tablet report, sending...
I20260619 12:00:00.048022 4399 ts_manager.cc:194] Registered new tserver with Master: fabc48447b03496289f4a52f8c0facc0 (127.0.203.193:40249)
I20260619 12:00:00.048837 4399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:36781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:00:00.073875 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40251
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:00.177917 4596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:00.178229 4596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:00.178269 4596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:00.180063 4596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:00.180220 4596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:00.180291 4596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:00.182897 4596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:00.183287 4596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:00.183588 4596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:00.189706 4601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:00.189869 4596 server_base.cc:1034] running on GCE node
W20260619 12:00:00.189845 4604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:00.189759 4602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:00.190240 4596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:00.190609 4596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:00.191848 4596 hybrid_clock.cc:648] HybridClock initialized: now 1781870400191832 us; error 79 us; skew 500 ppm
Jun 19 12:00:00 dist-test-slave-fbq8 krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:00.196023 4596 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:00:00.196808 4596 webserver.cc:460] Webserver started at http://127.0.203.194:35245/ using document root <none> and password file <none>
I20260619 12:00:00.197124 4596 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:00.197201 4596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:00.197345 4596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:00.198673 4596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "556aa8fac7fc4f12b44f94c355dc847c"
format_stamp: "Formatted at 2026-06-19 12:00:00 on dist-test-slave-fbq8"
server_key: "0f62a36d3b089162cc78184cde6cb03e"
server_key_iv: "d047c1d620ab64ae8f8b526212c76a78"
server_key_version: "encryptionkey@0"
I20260619 12:00:00.199009 4596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "556aa8fac7fc4f12b44f94c355dc847c"
format_stamp: "Formatted at 2026-06-19 12:00:00 on dist-test-slave-fbq8"
server_key: "0f62a36d3b089162cc78184cde6cb03e"
server_key_iv: "d047c1d620ab64ae8f8b526212c76a78"
server_key_version: "encryptionkey@0"
I20260619 12:00:00.200488 4596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:00.201588 4611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:00.201894 4596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:00.202008 4596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "556aa8fac7fc4f12b44f94c355dc847c"
format_stamp: "Formatted at 2026-06-19 12:00:00 on dist-test-slave-fbq8"
server_key: "0f62a36d3b089162cc78184cde6cb03e"
server_key_iv: "d047c1d620ab64ae8f8b526212c76a78"
server_key_version: "encryptionkey@0"
I20260619 12:00:00.202100 4596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:00.221299 4596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:00.224246 4596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:00.224412 4596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:00.224817 4596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:00.225209 4596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:00.225276 4596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:00.225324 4596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:00.225369 4596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:00.233191 4596 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:45335
I20260619 12:00:00.233376 4724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:45335 every 8 connection(s)
I20260619 12:00:00.233686 4596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:00:00 dist-test-slave-fbq8 krb5kdc[4354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:00.241580 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4596
I20260619 12:00:00.241683 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:00.241923 815 external_mini_cluster.cc:1428] Setting key 254889471122bb48e6523266f4469a14
I20260619 12:00:00.246447 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:00.234761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:41705 (local address 127.0.203.254:40251)
0619 12:00:00.234835 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:00.234837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:00.235107 (+ 270us) server_negotiation.cc:407] Connection header received
0619 12:00:00.236143 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:00.236144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:00.236158 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:00.236251 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:00.237246 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.237779 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.238920 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.239110 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.239170 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:00.241867 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:00.241869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:00.241870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:00.241878 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:00.244329 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:00.245032 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:00.245034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:00.245034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:00.245137 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:00.245515 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:00.245516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:00.245516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:00.245697 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:00.245770 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:00.246174 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 12:00:00.246293 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 12:00:00.246590 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:00.234706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40251 (local address 127.0.203.194:41705)
0619 12:00:00.234894 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:00.234940 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:00.235979 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:00.236406 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:00.236411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:00.236858 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:00.237082 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:00.237086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.237930 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.237931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:00.238795 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:00.238797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:00.239193 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:00.239194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:00.239325 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:00.239913 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:00.239922 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:00.241677 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:00.244485 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:00.244505 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:00.244510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:00.244930 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:00.245253 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:00.245254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:00.245255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:00.245403 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:00.245800 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:00.245807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:00.245976 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 12:00:00.246090 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 12:00:00.246260 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260619 12:00:00.247291 4725 heartbeater.cc:347] Connected to a master server at 127.0.203.254:40251
I20260619 12:00:00.247444 4725 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:00.247676 4725 heartbeater.cc:511] Master 127.0.203.254:40251 requested a full tablet report, sending...
I20260619 12:00:00.248116 4399 ts_manager.cc:194] Registered new tserver with Master: 556aa8fac7fc4f12b44f94c355dc847c (127.0.203.194:45335)
I20260619 12:00:00.248528 4399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:41705
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
I20260619 12:00:01.050246 4591 heartbeater.cc:503] Master 127.0.203.254:40251 was elected leader, sending a full tablet report...
I20260619 12:00:01.249509 4725 heartbeater.cc:503] Master 127.0.203.254:40251 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
W20260619 12:00:01.611963 815 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.203.195: real 1.369s user 0.000s sys 0.004s
I20260619 12:00:01.612524 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40251
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:01.707157 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:01.707399 4730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:01.707423 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:01.708603 4730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:01.708698 4730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:01.708748 4730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:01.710240 4730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:01.710462 4730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:01.710680 4730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:01.716490 4735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:01.716683 4730 server_base.cc:1034] running on GCE node
W20260619 12:00:01.716454 4738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:01.716569 4736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:01.717080 4730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:01.717430 4730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:01.718613 4730 hybrid_clock.cc:648] HybridClock initialized: now 1781870401718581 us; error 60 us; skew 500 ppm
Jun 19 12:00:01 dist-test-slave-fbq8 krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:01.726171 4730 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:00:01.726740 4730 webserver.cc:460] Webserver started at http://127.0.203.195:43817/ using document root <none> and password file <none>
I20260619 12:00:01.726992 4730 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:01.727064 4730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:01.727218 4730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:01.728547 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "db9e462c634a45a78eb3f029723b9edb"
format_stamp: "Formatted at 2026-06-19 12:00:01 on dist-test-slave-fbq8"
server_key: "654e4379a3788af15c352e8d7aeba9dd"
server_key_iv: "d07cee97738d65ee61eec768ec2e7140"
server_key_version: "encryptionkey@0"
I20260619 12:00:01.728993 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "db9e462c634a45a78eb3f029723b9edb"
format_stamp: "Formatted at 2026-06-19 12:00:01 on dist-test-slave-fbq8"
server_key: "654e4379a3788af15c352e8d7aeba9dd"
server_key_iv: "d07cee97738d65ee61eec768ec2e7140"
server_key_version: "encryptionkey@0"
I20260619 12:00:01.730819 4730 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:00:01.731978 4745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:01.732331 4730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:01.732529 4730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "db9e462c634a45a78eb3f029723b9edb"
format_stamp: "Formatted at 2026-06-19 12:00:01 on dist-test-slave-fbq8"
server_key: "654e4379a3788af15c352e8d7aeba9dd"
server_key_iv: "d07cee97738d65ee61eec768ec2e7140"
server_key_version: "encryptionkey@0"
I20260619 12:00:01.732636 4730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:01.773381 4730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:01.776412 4730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:01.776700 4730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:01.777184 4730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:01.777622 4730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:01.777684 4730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:01.777751 4730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:01.777781 4730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:01.785429 4730 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:41471
I20260619 12:00:01.785559 4858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:41471 every 8 connection(s)
I20260619 12:00:01.785997 4730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:01.789237 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4730
I20260619 12:00:01.789386 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:01.789614 815 external_mini_cluster.cc:1428] Setting key 4f6469538952a0db761f04a750c183f7
Jun 19 12:00:01 dist-test-slave-fbq8 krb5kdc[4354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:01.798004 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:01.787023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:44527 (local address 127.0.203.254:40251)
0619 12:00:01.787178 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:01.787180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:01.787270 (+ 90us) server_negotiation.cc:407] Connection header received
0619 12:00:01.788405 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:01.788408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:01.788441 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:01.788536 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:01.789605 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.790398 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.791725 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.791938 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.792006 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:01.794577 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:01.794579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:01.794580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:01.794587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:01.796303 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:01.796933 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:01.796934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:01.796934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:01.797001 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:01.797283 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:01.797283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:01.797284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:01.797416 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:01.797490 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:01.797801 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 12:00:01.797896 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260619 12:00:01.798229 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:01.786915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40251 (local address 127.0.203.195:44527)
0619 12:00:01.787111 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:01.787165 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:01.788218 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:01.788716 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:01.788721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:01.789142 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:01.789462 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:01.789468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.790511 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.790513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:01.791629 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:01.791633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.792031 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.792032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:01.792166 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:01.792801 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:01.792811 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:01.794464 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:01.796432 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:01.796445 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:01.796449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:01.796840 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:01.797069 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:01.797071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:01.797071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:01.797209 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:01.797491 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:01.797498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:01.797663 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 12:00:01.797770 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 12:00:01.797976 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":118,"threads_started":1}
I20260619 12:00:01.798817 4859 heartbeater.cc:347] Connected to a master server at 127.0.203.254:40251
I20260619 12:00:01.798935 4859 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:01.799188 4859 heartbeater.cc:511] Master 127.0.203.254:40251 requested a full tablet report, sending...
I20260619 12:00:01.799669 4399 ts_manager.cc:194] Registered new tserver with Master: db9e462c634a45a78eb3f029723b9edb (127.0.203.195:41471)
I20260619 12:00:01.800200 4399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:44527
I20260619 12:00:01.801877 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:00:01 dist-test-slave-fbq8 krb5kdc[4354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:00:01 dist-test-slave-fbq8 krb5kdc[4354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 12:00:01.913655 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:01.903242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.0.203.193:40249)
0619 12:00:01.903445 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:01.903451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:01.903468 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:00:01.903523 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:01.903525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:01.903543 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:01.903622 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:01.904323 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.905202 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.906153 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.906531 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.906609 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:01.909390 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:01.909407 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:01.909408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:01.909416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:01.912111 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:01.912679 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:01.912687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:01.912688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:01.912753 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:01.912975 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:01.912977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:01.912977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:01.913147 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:01.913350 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:01.913399 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 12:00:01.913508 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
Jun 19 12:00:01 dist-test-slave-fbq8 krb5kdc[4354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:01.923506 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:01.915400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.0.203.254:40251)
0619 12:00:01.915506 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:01.915509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:01.915520 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:01.915590 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:01.915591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:01.915611 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:01.915678 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:01.916253 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.916888 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.917606 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:01.917811 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:01.917904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:01.920440 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:01.920442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:01.920442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:01.920452 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:01.922156 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:01.922614 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:01.922615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:01.922616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:01.922678 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:01.922896 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:01.922897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:01.922897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:01.923112 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:01.923232 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:01.923320 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 12:00:01.923414 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 12:00:01.923929 4399 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41624
I20260619 12:00:01.924427 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4462
I20260619 12:00:01.929543 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4596
I20260619 12:00:01.933970 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4730
I20260619 12:00:01.939401 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4367
2026-06-19T12:00:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3475 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:02 dist-test-slave-fbq8 krb5kdc[4872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:02 dist-test-slave-fbq8 krb5kdc[4872](info): set up 2 sockets
Jun 19 12:00:02 dist-test-slave-fbq8 krb5kdc[4872](info): commencing operation
krb5kdc: starting...
W20260619 12:00:03.008797 815 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:00:03.144389 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:46785
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:46785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:03.230911 4885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:03.231144 4885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:03.231177 4885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:03.232290 4885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:03.232328 4885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:03.232342 4885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:03.232355 4885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:03.232371 4885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:03.233822 4885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:46785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:46785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:03.234025 4885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:03.234203 4885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:03.239686 4893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:03.239670 4890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:03.239647 4891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:03.239827 4885 server_base.cc:1034] running on GCE node
I20260619 12:00:03.240118 4885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:03.240469 4885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:03.241642 4885 hybrid_clock.cc:648] HybridClock initialized: now 1781870403241632 us; error 54 us; skew 500 ppm
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:03.244690 4885 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:00:03.245096 4885 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:03.245150 4885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:03.245301 4885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:03.246567 4885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "38d2fc8e504c4c869fdd219e22b1258f"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "64a728b8725668b0dd49a76f55997e1c"
server_key_iv: "dfad5d5001627a35bf0023866f5df593"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.246968 4885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "38d2fc8e504c4c869fdd219e22b1258f"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "64a728b8725668b0dd49a76f55997e1c"
server_key_iv: "dfad5d5001627a35bf0023866f5df593"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.248533 4885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:03.249528 4899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.249853 4885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:03.249989 4885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "38d2fc8e504c4c869fdd219e22b1258f"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "64a728b8725668b0dd49a76f55997e1c"
server_key_iv: "dfad5d5001627a35bf0023866f5df593"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.250082 4885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:03.266916 4885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:03.269973 4885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:03.270164 4885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:03.275138 4885 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:46785
I20260619 12:00:03.275185 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:46785 every 8 connection(s)
I20260619 12:00:03.275547 4885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:03.276902 4952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:03.279462 4952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f: Bootstrap starting.
I20260619 12:00:03.279775 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4885
I20260619 12:00:03.279925 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:03.280118 815 external_mini_cluster.cc:1428] Setting key 4e8d0292587c429af7638d457fb35436
I20260619 12:00:03.280313 4952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:03.280742 4952 log.cc:824] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:03.281616 4952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f: No bootstrap required, opened a new log
I20260619 12:00:03.283687 4952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d2fc8e504c4c869fdd219e22b1258f" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46785 } }
I20260619 12:00:03.283890 4952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:03.283936 4952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d2fc8e504c4c869fdd219e22b1258f, State: Initialized, Role: FOLLOWER
I20260619 12:00:03.284137 4952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d2fc8e504c4c869fdd219e22b1258f" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46785 } }
I20260619 12:00:03.284240 4952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:03.284288 4952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:03.284343 4952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:03.285179 4952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d2fc8e504c4c869fdd219e22b1258f" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46785 } }
I20260619 12:00:03.285320 4952 leader_election.cc:302] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38d2fc8e504c4c869fdd219e22b1258f; no voters:
I20260619 12:00:03.285559 4952 leader_election.cc:288] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:03.285593 4957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:03.285908 4952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:03.285895 4957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [term 1 LEADER]: Becoming Leader. State: Replica: 38d2fc8e504c4c869fdd219e22b1258f, State: Running, Role: LEADER
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:03.287499 4957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d2fc8e504c4c869fdd219e22b1258f" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46785 } }
I20260619 12:00:03.288232 4959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38d2fc8e504c4c869fdd219e22b1258f. Latest consensus state: current_term: 1 leader_uuid: "38d2fc8e504c4c869fdd219e22b1258f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d2fc8e504c4c869fdd219e22b1258f" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46785 } } }
I20260619 12:00:03.288832 4959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:03.288219 4958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38d2fc8e504c4c869fdd219e22b1258f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d2fc8e504c4c869fdd219e22b1258f" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 46785 } } }
I20260619 12:00:03.289145 4958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f [sys.catalog]: This master's current role is: LEADER
W20260619 12:00:03.289403 4971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:00:03.289469 4971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:00:03.292562 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.280859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.0.203.254:46785)
0619 12:00:03.281114 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.281124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.281156 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:00:03.281869 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.281880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.281938 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.282113 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:03.282799 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.283767 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.284790 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.285237 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.285367 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.288065 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:03.288070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:03.288071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:03.288083 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:03.290560 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.291109 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.291112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.291112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.291227 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.291501 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.291502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.291503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.291829 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:03.292005 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.292166 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.292321 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":125,"threads_started":1}
I20260619 12:00:03.292771 4974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:03.292961 4974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:03.295409 4974 catalog_manager.cc:1347] Generated new cluster ID: b7ec9b218a9b4a3badfd50f943ecdd52
I20260619 12:00:03.295480 4974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:03.320345 4974 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:03.321056 4974 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:03.332448 4974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f: Generated new TSK 0
I20260619 12:00:03.332700 4974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:00:03.373142 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46785
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:03.466121 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:03.466382 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:03.466425 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:03.467693 4978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:03.467767 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:03.467813 4978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:03.469585 4978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:03.469887 4978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:03.470149 4978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:03.476172 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:03.476172 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:03.476255 4978 server_base.cc:1034] running on GCE node
W20260619 12:00:03.476200 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:03.476847 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:03.477190 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:03.478389 4978 hybrid_clock.cc:648] HybridClock initialized: now 1781870403478370 us; error 57 us; skew 500 ppm
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:03.481382 4978 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:00:03.481819 4978 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:03.481899 4978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:03.482064 4978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:03.483184 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "a314b8f9840040bbb50e8544b598bdf6"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "78d0e590eb15bfaba2070d46d942893e"
server_key_iv: "a4b0ef2adf77bf2635d79422d0eff743"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.483553 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "a314b8f9840040bbb50e8544b598bdf6"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "78d0e590eb15bfaba2070d46d942893e"
server_key_iv: "a4b0ef2adf77bf2635d79422d0eff743"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.485078 4978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:03.486059 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.486332 4978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:03.486433 4978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "a314b8f9840040bbb50e8544b598bdf6"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "78d0e590eb15bfaba2070d46d942893e"
server_key_iv: "a4b0ef2adf77bf2635d79422d0eff743"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.486531 4978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:03.500363 4978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:03.502549 4978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:03.502703 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:03.502987 4978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:03.503325 4978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:03.503383 4978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.503429 4978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:03.503458 4978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.510417 4978 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:36111
I20260619 12:00:03.510564 5105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:36111 every 8 connection(s)
I20260619 12:00:03.510813 4978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:03.519465 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 4978
I20260619 12:00:03.519583 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:03.519842 815 external_mini_cluster.cc:1428] Setting key 52facfbac13f9581882d276cf368a314
I20260619 12:00:03.523336 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.512023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:60919 (local address 127.0.203.254:46785)
0619 12:00:03.512100 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.512103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.512380 (+ 277us) server_negotiation.cc:407] Connection header received
0619 12:00:03.513041 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.513042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.513053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.513137 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:03.514335 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.514933 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.516121 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.516333 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.516432 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.519827 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:03.519829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:03.519830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:03.519841 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:03.521668 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.522111 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.522112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.522113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.522179 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.522570 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.522571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.522571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.522715 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:03.522794 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.523122 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.523227 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:00:03.523584 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.511980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46785 (local address 127.0.203.193:60919)
0619 12:00:03.512215 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:03.512266 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:03.512935 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:03.513258 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:03.513263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:03.513623 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:03.514195 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.514200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.515070 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.515071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.516011 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.516014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.516468 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.516469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.516590 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.517291 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:03.517301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:03.519673 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:03.521771 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:03.521779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:03.521782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:03.522027 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:03.522249 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:03.522250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:03.522251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:03.522460 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:03.522799 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:03.522806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:03.522962 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 12:00:03.523090 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 12:00:03.523305 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":140,"threads_started":1}
I20260619 12:00:03.524245 5106 heartbeater.cc:347] Connected to a master server at 127.0.203.254:46785
I20260619 12:00:03.524369 5106 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:03.524576 5106 heartbeater.cc:511] Master 127.0.203.254:46785 requested a full tablet report, sending...
I20260619 12:00:03.525339 4916 ts_manager.cc:194] Registered new tserver with Master: a314b8f9840040bbb50e8544b598bdf6 (127.0.203.193:36111)
I20260619 12:00:03.526147 4916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:60919
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:00:03.553094 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46785
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:03.648491 5111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:03.648824 5111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:03.648865 5111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:03.650164 5111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:03.650238 5111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:03.650270 5111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:03.651912 5111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:03.652222 5111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:03.652484 5111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:00:03.658581 5111 server_base.cc:1034] running on GCE node
W20260619 12:00:03.658531 5116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:03.658515 5117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:03.658531 5119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:03.659271 5111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:03.659658 5111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:03.660830 5111 hybrid_clock.cc:648] HybridClock initialized: now 1781870403660811 us; error 40 us; skew 500 ppm
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:03.663852 5111 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:00:03.664217 5111 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:03.664268 5111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:03.664371 5111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:03.665633 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "91df771905754e3292e9880e78449362"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "7121b688f0054355e856729b895e7a89"
server_key_iv: "701e54e3ee0b79c69441d5efb0ab21ff"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.666065 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "91df771905754e3292e9880e78449362"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "7121b688f0054355e856729b895e7a89"
server_key_iv: "701e54e3ee0b79c69441d5efb0ab21ff"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.667641 5111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:03.668816 5125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.669147 5111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:03.669265 5111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "91df771905754e3292e9880e78449362"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "7121b688f0054355e856729b895e7a89"
server_key_iv: "701e54e3ee0b79c69441d5efb0ab21ff"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.669351 5111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:03.677809 5111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:03.680119 5111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:03.680240 5111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:03.680532 5111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:03.680918 5111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:03.680981 5111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.681006 5111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:03.681020 5111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.688531 5111 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:39147
I20260619 12:00:03.688621 5238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:39147 every 8 connection(s)
I20260619 12:00:03.688956 5111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:03.689553 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 5111
I20260619 12:00:03.689679 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:03.689918 815 external_mini_cluster.cc:1428] Setting key 5b0b9ca2da2f697fc27c58b1a37450a3
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
I20260619 12:00:03.700103 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.689980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:39563 (local address 127.0.203.254:46785)
0619 12:00:03.690049 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.690052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.690508 (+ 456us) server_negotiation.cc:407] Connection header received
0619 12:00:03.691193 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.691194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.691209 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.691271 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:03.692342 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.692959 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.694040 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.694258 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.694350 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.696882 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:03.696884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:03.696884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:03.696893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:03.698488 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.699016 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.699017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.699018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.699081 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.699377 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.699379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.699379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.699509 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:03.699592 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.699900 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.699988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 12:00:03.700371 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.689952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46785 (local address 127.0.203.194:39563)
0619 12:00:03.690365 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:03.690417 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:03.691067 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:03.691362 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:03.691366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:03.691709 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:03.692225 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.692230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.693077 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.693078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.693923 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.693925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.694360 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.694361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.694494 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.695119 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:03.695131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:03.696764 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:03.698604 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:03.698613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:03.698615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:03.698893 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:03.699173 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:03.699174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:03.699175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:03.699303 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:03.699581 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:03.699588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:03.699747 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 12:00:03.699876 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 12:00:03.700101 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":127,"threads_started":1}
I20260619 12:00:03.700985 5239 heartbeater.cc:347] Connected to a master server at 127.0.203.254:46785
I20260619 12:00:03.701146 5239 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:03.701355 5239 heartbeater.cc:511] Master 127.0.203.254:46785 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
I20260619 12:00:03.701915 4916 ts_manager.cc:194] Registered new tserver with Master: 91df771905754e3292e9880e78449362 (127.0.203.194:39147)
I20260619 12:00:03.702456 4916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:39563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:00:03.721133 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46785
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:34295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:03.816526 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:03.816870 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:03.816897 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:03.818104 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:03.818142 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:03.818169 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:03.819664 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:34295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:03.819880 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:03.820092 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:03.825987 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:03.826040 5252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:03.826341 5244 server_base.cc:1034] running on GCE node
W20260619 12:00:03.825992 5249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:03.827049 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:03.827499 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:03.828732 5244 hybrid_clock.cc:648] HybridClock initialized: now 1781870403828699 us; error 57 us; skew 500 ppm
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:03.831674 5244 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:00:03.832145 5244 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:03.832201 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:03.832360 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:03.833681 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "04ee4be43a2d4d8a9a0755ed89729b8d"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "e6caed27a56808f1b5e26a7da00ef23e"
server_key_iv: "3e99c27feaa1d0d404b881d3ea36d6af"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.834064 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "04ee4be43a2d4d8a9a0755ed89729b8d"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "e6caed27a56808f1b5e26a7da00ef23e"
server_key_iv: "3e99c27feaa1d0d404b881d3ea36d6af"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.835562 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:03.836453 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.836706 5244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:03.836812 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "04ee4be43a2d4d8a9a0755ed89729b8d"
format_stamp: "Formatted at 2026-06-19 12:00:03 on dist-test-slave-fbq8"
server_key: "e6caed27a56808f1b5e26a7da00ef23e"
server_key_iv: "3e99c27feaa1d0d404b881d3ea36d6af"
server_key_version: "encryptionkey@0"
I20260619 12:00:03.836898 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:03.858860 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:03.861289 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:03.861464 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:03.861745 5244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:03.862114 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:03.862174 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.862221 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:03.862246 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:03.869954 5244 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:36305
I20260619 12:00:03.870030 5371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:36305 every 8 connection(s)
I20260619 12:00:03.870352 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:00:03 dist-test-slave-fbq8 krb5kdc[4872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:03.878325 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 5244
I20260619 12:00:03.878448 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:03.878698 815 external_mini_cluster.cc:1428] Setting key cce0c70d8f4222db9fc840578a24d814
I20260619 12:00:03.881918 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.871224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:46785 (local address 127.0.203.195:33327)
0619 12:00:03.871441 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:03.871490 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:03.872115 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:03.872558 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:03.872562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:03.872966 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:03.873611 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.873617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.874367 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.874368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.875281 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.875284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.875726 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.875727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.875846 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.876478 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:03.876487 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:03.878028 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:03.880237 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:03.880244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:03.880245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:03.880506 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:03.880847 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:03.880848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:03.880848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:03.880976 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:03.881285 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:03.881291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:03.881444 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 12:00:03.881556 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 12:00:03.881699 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":130,"threads_started":1}
I20260619 12:00:03.881929 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.871278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:33327 (local address 127.0.203.254:46785)
0619 12:00:03.871330 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.871333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.871594 (+ 261us) server_negotiation.cc:407] Connection header received
0619 12:00:03.872262 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.872263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.872274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.872409 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:03.873714 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.874250 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.875387 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.875607 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.875680 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.878145 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:03.878147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:03.878147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:03.878156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:03.880124 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.880623 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.880624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.880624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.880764 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.881058 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.881059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.881059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.881181 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:03.881242 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.881645 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.881854 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 12:00:03.882540 5372 heartbeater.cc:347] Connected to a master server at 127.0.203.254:46785
I20260619 12:00:03.882659 5372 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:03.882875 5372 heartbeater.cc:511] Master 127.0.203.254:46785 requested a full tablet report, sending...
I20260619 12:00:03.883320 4916 ts_manager.cc:194] Registered new tserver with Master: 04ee4be43a2d4d8a9a0755ed89729b8d (127.0.203.195:36305)
I20260619 12:00:03.883774 4916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:33327
I20260619 12:00:03.891428 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:00:03.900350 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.893027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.0.203.254:46785)
0619 12:00:03.893094 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.893102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.893287 (+ 185us) server_negotiation.cc:407] Connection header received
0619 12:00:03.893317 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.893318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.893330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.893412 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:03.894020 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.894714 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.895796 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.895993 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.896068 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.897465 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:03.897467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:03.897467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:03.897474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:03.899090 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.899505 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.899506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.899506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.899582 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:03.899828 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:03.899828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:03.899829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:03.899990 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:03.900078 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.900191 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.900264 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 12:00:03.902603 4916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:03.904372 4916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:03.912153 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.907931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.203.193:36111)
0619 12:00:03.908344 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.908350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.908362 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:03.908405 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.908407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.908425 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.908512 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:03.908691 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.909823 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.911138 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.911683 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.911776 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.911846 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.911983 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.912043 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":83,"threads_started":1}
I20260619 12:00:03.912139 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.907457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:36111 (local address 127.0.0.1:38374)
0619 12:00:03.907852 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:03.907898 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:03.907970 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:03.908509 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:03.908510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:03.908513 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:03.908569 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.908571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.909940 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.909941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.911036 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.911038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.911775 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.911776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.911859 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.911873 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 12:00:03.911967 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:00:03.912039 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":81,"threads_started":1}
I20260619 12:00:03.912389 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.907691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:39147 (local address 127.0.0.1:57648)
0619 12:00:03.908122 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:03.908131 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:03.908168 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:03.908907 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:03.908908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:03.908911 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:03.908954 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.908955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.910192 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.910193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.911138 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.911141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.912070 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.912071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.912157 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.912171 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 12:00:03.912205 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 12:00:03.912269 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":6,"thread_start_us":102,"threads_started":1}
I20260619 12:00:03.912557 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.907457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36305 (local address 127.0.0.1:35976)
0619 12:00:03.907883 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:03.907898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:03.907970 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:03.908387 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:03.908392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:03.908395 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:03.908498 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.908500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.909606 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.909607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.910991 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:03.910994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.912067 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.912069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:03.912157 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.912169 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 12:00:03.912205 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 12:00:03.912270 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":46,"thread_start_us":82,"threads_started":1}
I20260619 12:00:03.912757 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.907659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.0.203.195:36305)
0619 12:00:03.908157 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.908163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.908178 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:00:03.908229 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.908231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.908253 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.908365 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:03.908591 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.909500 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.911115 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.911955 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.912325 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.912475 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.912508 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.912556 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":150,"threads_started":1}
I20260619 12:00:03.913115 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:03.907855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.0.203.194:39147)
0619 12:00:03.908713 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:03.908717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:03.908729 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:03.908770 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:03.908772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:03.908788 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:03.908899 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:03.909041 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.910034 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.911239 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:03.911954 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:03.912792 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:03.912864 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:03.912891 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 12:00:03.912943 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":93,"threads_started":1}
I20260619 12:00:03.913410 5040 tablet_service.cc:1467] Processing CreateTablet for tablet 572c2f86a196427089340aeef24374be (DEFAULT_TABLE table=test-table [id=8cb97ec71d9043ce8f055c5492863b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:03.913786 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572c2f86a196427089340aeef24374be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:03.917543 5389 tablet_bootstrap.cc:492] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: Bootstrap starting.
I20260619 12:00:03.917505 5306 tablet_service.cc:1467] Processing CreateTablet for tablet 572c2f86a196427089340aeef24374be (DEFAULT_TABLE table=test-table [id=8cb97ec71d9043ce8f055c5492863b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:03.917665 5173 tablet_service.cc:1467] Processing CreateTablet for tablet 572c2f86a196427089340aeef24374be (DEFAULT_TABLE table=test-table [id=8cb97ec71d9043ce8f055c5492863b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:03.917968 5173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572c2f86a196427089340aeef24374be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:03.917968 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572c2f86a196427089340aeef24374be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:03.918251 5389 tablet_bootstrap.cc:654] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:03.918968 5389 log.cc:824] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:03.920035 5389 tablet_bootstrap.cc:492] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: No bootstrap required, opened a new log
I20260619 12:00:03.920092 5389 ts_tablet_manager.cc:1397] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:00:03.921998 5391 tablet_bootstrap.cc:492] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Bootstrap starting.
I20260619 12:00:03.922152 5392 tablet_bootstrap.cc:492] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Bootstrap starting.
I20260619 12:00:03.922770 5392 tablet_bootstrap.cc:654] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:03.922833 5391 tablet_bootstrap.cc:654] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:03.923461 5392 log.cc:824] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:03.923441 5389 raft_consensus.cc:348] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:03.923918 5389 raft_consensus.cc:374] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:03.923918 5391 log.cc:824] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:03.923965 5389 raft_consensus.cc:729] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a314b8f9840040bbb50e8544b598bdf6, State: Initialized, Role: FOLLOWER
I20260619 12:00:03.924152 5389 consensus_queue.cc:260] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:03.924505 5392 tablet_bootstrap.cc:492] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: No bootstrap required, opened a new log
I20260619 12:00:03.924507 5106 heartbeater.cc:503] Master 127.0.203.254:46785 was elected leader, sending a full tablet report...
I20260619 12:00:03.924516 5389 ts_tablet_manager.cc:1428] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 12:00:03.924683 5392 ts_tablet_manager.cc:1397] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:00:03.924758 5391 tablet_bootstrap.cc:492] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: No bootstrap required, opened a new log
I20260619 12:00:03.924831 5391 ts_tablet_manager.cc:1397] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:00:03.926872 5391 raft_consensus.cc:348] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:03.926872 5392 raft_consensus.cc:348] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:03.927047 5392 raft_consensus.cc:374] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:03.927047 5391 raft_consensus.cc:374] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:03.927084 5391 raft_consensus.cc:729] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ee4be43a2d4d8a9a0755ed89729b8d, State: Initialized, Role: FOLLOWER
I20260619 12:00:03.927084 5392 raft_consensus.cc:729] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91df771905754e3292e9880e78449362, State: Initialized, Role: FOLLOWER
I20260619 12:00:03.927222 5391 consensus_queue.cc:260] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:03.927222 5392 consensus_queue.cc:260] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:03.927551 5392 ts_tablet_manager.cc:1428] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:00:03.927551 5391 ts_tablet_manager.cc:1428] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:00:03.927788 5372 heartbeater.cc:503] Master 127.0.203.254:46785 was elected leader, sending a full tablet report...
I20260619 12:00:03.927857 5239 heartbeater.cc:503] Master 127.0.203.254:46785 was elected leader, sending a full tablet report...
W20260619 12:00:03.939702 5240 tablet.cc:2369] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:00:04.011826 5107 tablet.cc:2369] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:00:04.087467 5396 raft_consensus.cc:482] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:04.087684 5396 raft_consensus.cc:504] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:04.088140 5396 leader_election.cc:288] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a314b8f9840040bbb50e8544b598bdf6 (127.0.203.193:36111), 91df771905754e3292e9880e78449362 (127.0.203.194:39147)
I20260619 12:00:04.091694 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.088462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:36111 (local address 127.0.203.195:56775)
0619 12:00:04.088564 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.088599 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.088727 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.088994 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.088996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.089000 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:04.089075 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:04.089077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.089854 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.089855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:04.090845 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:04.090848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.091395 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.091396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:04.091485 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:04.091488 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.091561 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.091610 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 12:00:04.091691 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.088547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:56775 (local address 127.0.203.193:36111)
0619 12:00:04.088632 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.088659 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.088698 (+ 39us) server_negotiation.cc:407] Connection header received
0619 12:00:04.088874 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.088876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.088892 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.088970 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:04.089183 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.089771 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.090932 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.091314 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.091399 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:04.091410 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.091568 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.091606 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:00:04.091863 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.088614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:39147 (local address 127.0.203.195:35175)
0619 12:00:04.088891 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.088901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.088992 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.089146 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.089147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.089150 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:04.089258 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:04.089260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.090107 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.090108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:04.090929 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:04.090930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.091553 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.091554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:04.091635 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:04.091640 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.091726 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.091771 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":199,"threads_started":1}
I20260619 12:00:04.091851 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.088728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:35175 (local address 127.0.203.194:39147)
0619 12:00:04.088811 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.088813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.088978 (+ 165us) server_negotiation.cc:407] Connection header received
0619 12:00:04.089054 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.089055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.089067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.089142 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:04.089338 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.090009 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.091021 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.091467 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.091555 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:04.091566 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.091733 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.091774 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:00:04.092420 5060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572c2f86a196427089340aeef24374be" candidate_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a314b8f9840040bbb50e8544b598bdf6" is_pre_election: true
I20260619 12:00:04.092552 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572c2f86a196427089340aeef24374be" candidate_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91df771905754e3292e9880e78449362" is_pre_election: true
I20260619 12:00:04.092610 5060 raft_consensus.cc:2454] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04ee4be43a2d4d8a9a0755ed89729b8d in term 0.
I20260619 12:00:04.092753 5193 raft_consensus.cc:2454] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04ee4be43a2d4d8a9a0755ed89729b8d in term 0.
I20260619 12:00:04.092916 5261 leader_election.cc:302] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04ee4be43a2d4d8a9a0755ed89729b8d, a314b8f9840040bbb50e8544b598bdf6; no voters:
I20260619 12:00:04.093086 5396 raft_consensus.cc:2781] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:04.093145 5396 raft_consensus.cc:482] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:04.093174 5396 raft_consensus.cc:3037] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:04.094022 5396 raft_consensus.cc:504] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:04.094169 5396 leader_election.cc:288] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [CANDIDATE]: Term 1 election: Requested vote from peers a314b8f9840040bbb50e8544b598bdf6 (127.0.203.193:36111), 91df771905754e3292e9880e78449362 (127.0.203.194:39147)
I20260619 12:00:04.094398 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572c2f86a196427089340aeef24374be" candidate_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91df771905754e3292e9880e78449362"
I20260619 12:00:04.094398 5060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572c2f86a196427089340aeef24374be" candidate_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a314b8f9840040bbb50e8544b598bdf6"
I20260619 12:00:04.094472 5193 raft_consensus.cc:3037] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:04.094499 5060 raft_consensus.cc:3037] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:04.095224 5193 raft_consensus.cc:2454] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04ee4be43a2d4d8a9a0755ed89729b8d in term 1.
I20260619 12:00:04.095391 5261 leader_election.cc:302] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04ee4be43a2d4d8a9a0755ed89729b8d, 91df771905754e3292e9880e78449362; no voters:
I20260619 12:00:04.095394 5060 raft_consensus.cc:2454] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04ee4be43a2d4d8a9a0755ed89729b8d in term 1.
I20260619 12:00:04.095500 5396 raft_consensus.cc:2781] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:04.095697 5396 raft_consensus.cc:686] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 1 LEADER]: Becoming Leader. State: Replica: 04ee4be43a2d4d8a9a0755ed89729b8d, State: Running, Role: LEADER
I20260619 12:00:04.095944 5396 consensus_queue.cc:237] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } }
I20260619 12:00:04.097087 4916 catalog_manager.cc:5495] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 04ee4be43a2d4d8a9a0755ed89729b8d (127.0.203.195). New cstate: current_term: 1 leader_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04ee4be43a2d4d8a9a0755ed89729b8d" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 } health_report { overall_health: UNKNOWN } } }
W20260619 12:00:04.121030 5373 tablet.cc:2369] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:00:04.143585 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.140962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.0.203.195:36305)
0619 12:00:04.141071 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.141074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.141115 (+ 41us) server_negotiation.cc:407] Connection header received
0619 12:00:04.141231 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.141232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.141251 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.141332 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:04.141485 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.141963 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.142761 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:04.142958 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:04.143033 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:04.143222 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:04.143327 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:04.143424 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.143443 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.143463 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 12:00:04.145980 5193 raft_consensus.cc:1261] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 1 FOLLOWER]: Refusing update from remote peer 04ee4be43a2d4d8a9a0755ed89729b8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:04.146016 5060 raft_consensus.cc:1261] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 1 FOLLOWER]: Refusing update from remote peer 04ee4be43a2d4d8a9a0755ed89729b8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:04.146584 5399 consensus_queue.cc:1035] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91df771905754e3292e9880e78449362" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:04.146765 5396 consensus_queue.cc:1035] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a314b8f9840040bbb50e8544b598bdf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:04.149976 5403 mvcc.cc:205] Tried to move back new op lower bound from 7298541175377551360 to 7298541175177908224. Current Snapshot: MvccSnapshot[applied={T|T < 7298541175377551360}]
I20260619 12:00:04.149601 5402 mvcc.cc:205] Tried to move back new op lower bound from 7298541175377551360 to 7298541175177908224. Current Snapshot: MvccSnapshot[applied={T|T < 7298541175377551360}]
I20260619 12:00:04.153806 4915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:04.153955 4915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:04.154924 4915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38d2fc8e504c4c869fdd219e22b1258f: Sending DeleteTablet for 3 replicas of tablet 572c2f86a196427089340aeef24374be
I20260619 12:00:04.155360 5173 tablet_service.cc:1514] Processing DeleteTablet for tablet 572c2f86a196427089340aeef24374be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:04 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:57648
I20260619 12:00:04.155485 5306 tablet_service.cc:1514] Processing DeleteTablet for tablet 572c2f86a196427089340aeef24374be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:04 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:35976
I20260619 12:00:04.155663 5040 tablet_service.cc:1514] Processing DeleteTablet for tablet 572c2f86a196427089340aeef24374be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:04 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:38374
I20260619 12:00:04.155705 5412 tablet_replica.cc:330] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: stopping tablet replica
I20260619 12:00:04.155853 5412 raft_consensus.cc:2229] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:04.155884 5414 tablet_replica.cc:330] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6: stopping tablet replica
I20260619 12:00:04.155977 5412 raft_consensus.cc:2258] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:04.156004 5414 raft_consensus.cc:2229] T 572c2f86a196427089340aeef24374be P a314b8f9840040bbb50e8544b598bdf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:04.156461 5413 tablet_replica.cc:330] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: stopping tablet replica
I20260619 12:00:04.156529 5412 ts_tablet_manager.cc:1905] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:04.156689 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4978
I20260619 12:00:04.157783 5413 raft_consensus.cc:2229] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:00:04.157998 5413 raft_consensus.cc:2258] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:04.158532 5413 ts_tablet_manager.cc:1905] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:04.160259 5413 ts_tablet_manager.cc:1918] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:04.160355 5413 log.cc:1196] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/572c2f86a196427089340aeef24374be
I20260619 12:00:04.160715 5413 ts_tablet_manager.cc:1939] T 572c2f86a196427089340aeef24374be P 04ee4be43a2d4d8a9a0755ed89729b8d: Deleting consensus metadata
I20260619 12:00:04.162747 5412 ts_tablet_manager.cc:1918] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:04.162865 5412 log.cc:1196] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/572c2f86a196427089340aeef24374be
I20260619 12:00:04.163200 5412 ts_tablet_manager.cc:1939] T 572c2f86a196427089340aeef24374be P 91df771905754e3292e9880e78449362: Deleting consensus metadata
I20260619 12:00:04.163704 4902 catalog_manager.cc:4841] TS 91df771905754e3292e9880e78449362 (127.0.203.194:39147): tablet 572c2f86a196427089340aeef24374be (table test-table [id=8cb97ec71d9043ce8f055c5492863b87]) successfully deleted
I20260619 12:00:04.163856 4903 catalog_manager.cc:4841] TS 04ee4be43a2d4d8a9a0755ed89729b8d (127.0.203.195:36305): tablet 572c2f86a196427089340aeef24374be (table test-table [id=8cb97ec71d9043ce8f055c5492863b87]) successfully deleted
W20260619 12:00:04.164836 4902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:36111 (error 108)
I20260619 12:00:04.165402 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 5111
I20260619 12:00:04.165480 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.165247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:36111 (local address 127.0.0.1:38384)
0619 12:00:04.165347 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.165424 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 12:00:04.168525 4902 catalog_manager.cc:4584] TS a314b8f9840040bbb50e8544b598bdf6 (127.0.203.193:36111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 572c2f86a196427089340aeef24374be: Network error: Client connection negotiation failed: client connection to 127.0.203.193:36111: connect: Connection refused (error 111)
I20260619 12:00:04.172165 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 5244
I20260619 12:00:04.178408 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 4885
2026-06-19T12:00:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2238 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T12:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:04Z Disabled control of system clock
I20260619 12:00:04.208866 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36649
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:00:04.297541 5419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:04.297951 5419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:04.297992 5419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:04.299391 5419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:04.299517 5419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:04.299541 5419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:04.299567 5419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:04.299599 5419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:04.301298 5419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:36649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:04.301656 5419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:04.302003 5419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:04.307732 5424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:04.307834 5425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:04.308012 5419 server_base.cc:1034] running on GCE node
W20260619 12:00:04.307740 5427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:04.308419 5419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:04.308859 5419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:04.310056 5419 hybrid_clock.cc:648] HybridClock initialized: now 1781870404310042 us; error 60 us; skew 500 ppm
I20260619 12:00:04.311548 5419 webserver.cc:460] Webserver started at http://127.0.203.254:32823/ using document root <none> and password file <none>
I20260619 12:00:04.311836 5419 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:04.311909 5419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:04.312081 5419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:04.313397 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "7ed4184a04fb4e3f9e628d9e0a225898"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "0cab3583692b9a9f3b149b39c734a040"
server_key_iv: "006eb300539722a046abaf203f8d77fe"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.313812 5419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "7ed4184a04fb4e3f9e628d9e0a225898"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "0cab3583692b9a9f3b149b39c734a040"
server_key_iv: "006eb300539722a046abaf203f8d77fe"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.315341 5419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:04.316337 5433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.316751 5419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:04.316898 5419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "7ed4184a04fb4e3f9e628d9e0a225898"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "0cab3583692b9a9f3b149b39c734a040"
server_key_iv: "006eb300539722a046abaf203f8d77fe"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.316990 5419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:04.320580 5419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:04.320916 5419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:04.321070 5419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:04.326105 5419 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:36649
I20260619 12:00:04.326182 5485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:36649 every 8 connection(s)
I20260619 12:00:04.326592 5419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:04.327418 5486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:04.330466 5486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898: Bootstrap starting.
I20260619 12:00:04.331069 5486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:04.331388 5486 log.cc:824] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:04.332284 5486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898: No bootstrap required, opened a new log
I20260619 12:00:04.334076 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 5419
I20260619 12:00:04.334192 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:04.334136 5486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36649 } }
I20260619 12:00:04.334347 5486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:04.334370 815 external_mini_cluster.cc:1428] Setting key 26811fa94301b0b5113eb113ed1e8a6a
I20260619 12:00:04.334385 5486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed4184a04fb4e3f9e628d9e0a225898, State: Initialized, Role: FOLLOWER
I20260619 12:00:04.334561 5486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36649 } }
I20260619 12:00:04.334648 5486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:04.334671 5486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:04.334730 5486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:04.335479 5486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36649 } }
I20260619 12:00:04.335572 5486 leader_election.cc:302] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ed4184a04fb4e3f9e628d9e0a225898; no voters:
I20260619 12:00:04.335757 5486 leader_election.cc:288] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:04.336120 5486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:04.335789 5491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:04.336357 5491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [term 1 LEADER]: Becoming Leader. State: Replica: 7ed4184a04fb4e3f9e628d9e0a225898, State: Running, Role: LEADER
I20260619 12:00:04.336539 5491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36649 } }
I20260619 12:00:04.337359 5492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36649 } } }
I20260619 12:00:04.337467 5492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:04.338121 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.335058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.203.254:36649)
0619 12:00:04.335341 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.335360 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.335389 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:00:04.335518 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.335525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.335600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.335761 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.337410 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.337414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.337415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.337419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.337470 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.337486 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.337797 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.337832 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.337896 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":124,"threads_started":1}
I20260619 12:00:04.337864 5493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ed4184a04fb4e3f9e628d9e0a225898. Latest consensus state: current_term: 1 leader_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed4184a04fb4e3f9e628d9e0a225898" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36649 } } }
I20260619 12:00:04.338181 5493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:04.338379 5501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:04.338953 5501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:04.340390 5501 catalog_manager.cc:1347] Generated new cluster ID: 800cf7f2d85141eeae984dbe9cb69762
I20260619 12:00:04.340446 5501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:04.385102 5501 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:04.385381 5501 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:04.389015 5501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898: Generated new TSK 0
I20260619 12:00:04.389217 5501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:00:04.391004 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36649
--builtin_ntp_servers=127.0.203.212:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:00:04.473699 5510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:04.473948 5510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:04.473973 5510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:04.475100 5510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:04.475143 5510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:04.475169 5510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:04.476584 5510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:04.476893 5510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:04.477237 5510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:04.482990 5518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:04.482957 5515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:04.483088 5516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:04.483400 5510 server_base.cc:1034] running on GCE node
I20260619 12:00:04.483587 5510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:04.483807 5510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:04.484977 5510 hybrid_clock.cc:648] HybridClock initialized: now 1781870404484946 us; error 49 us; skew 500 ppm
I20260619 12:00:04.486354 5510 webserver.cc:460] Webserver started at http://127.0.203.193:33521/ using document root <none> and password file <none>
I20260619 12:00:04.486575 5510 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:04.486622 5510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:04.486725 5510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:04.487785 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "7609da890d864f14a7df0b542f1ed148"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "af994edf0808e70d67db29663806a5fb"
server_key_iv: "801f48d53404c04fbd21594b1894b027"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.488107 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "7609da890d864f14a7df0b542f1ed148"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "af994edf0808e70d67db29663806a5fb"
server_key_iv: "801f48d53404c04fbd21594b1894b027"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.489560 5510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:04.490435 5524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.490690 5510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:04.490806 5510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "7609da890d864f14a7df0b542f1ed148"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "af994edf0808e70d67db29663806a5fb"
server_key_iv: "801f48d53404c04fbd21594b1894b027"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.490896 5510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:04.494676 5510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:04.494935 5510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:04.495035 5510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:04.495234 5510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:04.495525 5510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:04.495558 5510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.495579 5510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:04.495591 5510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.502431 5510 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:40835
I20260619 12:00:04.502540 5637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:40835 every 8 connection(s)
I20260619 12:00:04.502902 5510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:04.506165 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 5510
I20260619 12:00:04.506287 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:04.506490 815 external_mini_cluster.cc:1428] Setting key 85b364f52222cd274df1034c122c8fd1
I20260619 12:00:04.506502 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.503803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:48465 (local address 127.0.203.254:36649)
0619 12:00:04.503900 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.503903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.504167 (+ 264us) server_negotiation.cc:407] Connection header received
0619 12:00:04.504706 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.504707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.504719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.504800 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.506135 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.506136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.506136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.506140 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.506158 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.506163 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.506223 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.506389 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.506420 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:00:04.506934 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.503792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36649 (local address 127.0.203.193:48465)
0619 12:00:04.504026 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.504069 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.504552 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.504872 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.504876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.504884 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.505903 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.505903 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.505984 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.506024 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.506243 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.506248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.506252 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.506626 (+ 374us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.506711 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":132,"threads_started":1}
I20260619 12:00:04.507645 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36649
--builtin_ntp_servers=127.0.203.212:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 12:00:04.507987 5638 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36649
I20260619 12:00:04.508124 5638 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:04.508304 5638 heartbeater.cc:511] Master 127.0.203.254:36649 requested a full tablet report, sending...
I20260619 12:00:04.508996 5450 ts_manager.cc:194] Registered new tserver with Master: 7609da890d864f14a7df0b542f1ed148 (127.0.203.193:40835)
W20260619 12:00:04.597237 5641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:04.597522 5641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:04.597558 5641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:04.598718 5641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:04.598783 5641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:04.598829 5641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:04.600195 5641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:04.600487 5641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:04.600832 5641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:04.606521 5649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:04.606595 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:04.606585 5646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:04.606819 5641 server_base.cc:1034] running on GCE node
I20260619 12:00:04.607061 5641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:04.607358 5641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:04.608527 5641 hybrid_clock.cc:648] HybridClock initialized: now 1781870404608511 us; error 43 us; skew 500 ppm
I20260619 12:00:04.610045 5641 webserver.cc:460] Webserver started at http://127.0.203.194:46377/ using document root <none> and password file <none>
I20260619 12:00:04.610364 5641 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:04.610443 5641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:04.610621 5641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:04.611800 5641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "3193a4b597e04393bc1af4448de2c694"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "7618157fdc2e828839e4b56166459c1b"
server_key_iv: "0001b106c95cdf0e8b24925b5af444b2"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.612205 5641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "3193a4b597e04393bc1af4448de2c694"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "7618157fdc2e828839e4b56166459c1b"
server_key_iv: "0001b106c95cdf0e8b24925b5af444b2"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.613737 5641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:04.614643 5655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.615018 5641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:04.615192 5641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "3193a4b597e04393bc1af4448de2c694"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "7618157fdc2e828839e4b56166459c1b"
server_key_iv: "0001b106c95cdf0e8b24925b5af444b2"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.615329 5641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:04.619300 5641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:04.619674 5641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:04.619853 5641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:04.620088 5641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:04.620419 5641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:04.620474 5641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.620517 5641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:04.620553 5641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.628518 5641 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:33607
I20260619 12:00:04.628572 5768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:33607 every 8 connection(s)
I20260619 12:00:04.628968 5641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:04.632944 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.630058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:45463 (local address 127.0.203.254:36649)
0619 12:00:04.630160 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.630162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.630368 (+ 206us) server_negotiation.cc:407] Connection header received
0619 12:00:04.630903 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.630904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.630910 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.630968 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.632426 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.632427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.632428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.632433 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.632455 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.632460 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.632537 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.632828 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.632848 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 12:00:04.633087 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.629978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36649 (local address 127.0.203.194:45463)
0619 12:00:04.630230 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.630273 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.630796 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.631114 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.631119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.631130 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.632202 (+ 1072us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.632203 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.632236 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.632270 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.632613 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.632619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.632622 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.632797 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.632859 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":120,"threads_started":1}
I20260619 12:00:04.633797 5769 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36649
I20260619 12:00:04.633903 5769 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:04.633908 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 5641
I20260619 12:00:04.634012 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:04.634044 5769 heartbeater.cc:511] Master 127.0.203.254:36649 requested a full tablet report, sending...
I20260619 12:00:04.634181 815 external_mini_cluster.cc:1428] Setting key 5c323f55f604a8a213ce9f4b4c6fb631
I20260619 12:00:04.634506 5450 ts_manager.cc:194] Registered new tserver with Master: 3193a4b597e04393bc1af4448de2c694 (127.0.203.194:33607)
I20260619 12:00:04.635442 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36649
--builtin_ntp_servers=127.0.203.212:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:00:04.722041 5772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:04.722306 5772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:04.722333 5772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:04.723702 5772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:04.723765 5772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:04.723803 5772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:04.725359 5772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:04.725660 5772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:04.726011 5772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:04.731933 5780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:04.732059 5777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:04.732117 5772 server_base.cc:1034] running on GCE node
W20260619 12:00:04.731940 5778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:04.732438 5772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:04.732793 5772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:04.733975 5772 hybrid_clock.cc:648] HybridClock initialized: now 1781870404733956 us; error 42 us; skew 500 ppm
I20260619 12:00:04.735899 5772 webserver.cc:460] Webserver started at http://127.0.203.195:35281/ using document root <none> and password file <none>
I20260619 12:00:04.736238 5772 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:04.736295 5772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:04.736441 5772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:04.737709 5772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "ab7fb5293a5044b28931cc4a8b8b98b6"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "ed7c6f4552149eaaad8e9b2cb7dcd012"
server_key_iv: "0f6006ca6e3cc59dd25ddacc877ec952"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.738044 5772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "ab7fb5293a5044b28931cc4a8b8b98b6"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "ed7c6f4552149eaaad8e9b2cb7dcd012"
server_key_iv: "0f6006ca6e3cc59dd25ddacc877ec952"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.739388 5772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:00:04.740185 5786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.740437 5772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:04.740554 5772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "ab7fb5293a5044b28931cc4a8b8b98b6"
format_stamp: "Formatted at 2026-06-19 12:00:04 on dist-test-slave-fbq8"
server_key: "ed7c6f4552149eaaad8e9b2cb7dcd012"
server_key_iv: "0f6006ca6e3cc59dd25ddacc877ec952"
server_key_version: "encryptionkey@0"
I20260619 12:00:04.740643 5772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:04.744350 5772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:04.744706 5772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:04.744895 5772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:04.745128 5772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:04.745453 5772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:04.745507 5772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.745549 5772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:04.745576 5772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:04.752795 5772 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:43245
I20260619 12:00:04.752885 5899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:43245 every 8 connection(s)
I20260619 12:00:04.753260 5772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:04.756789 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.754136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:57905 (local address 127.0.203.254:36649)
0619 12:00:04.754245 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.754249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.754491 (+ 242us) server_negotiation.cc:407] Connection header received
0619 12:00:04.755115 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.755116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.755124 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.755186 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.756458 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.756459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.756460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.756464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.756481 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.756486 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.756550 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.756679 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.756702 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:00:04.757036 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.754090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36649 (local address 127.0.203.195:57905)
0619 12:00:04.754337 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.754379 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.754995 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.755275 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.755280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.755290 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.756283 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.756284 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.756316 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.756350 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.756548 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.756553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.756557 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.756696 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.756765 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":102,"threads_started":1}
I20260619 12:00:04.757674 5900 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36649
I20260619 12:00:04.757790 5900 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:04.757964 5900 heartbeater.cc:511] Master 127.0.203.254:36649 requested a full tablet report, sending...
I20260619 12:00:04.758666 5449 ts_manager.cc:194] Registered new tserver with Master: ab7fb5293a5044b28931cc4a8b8b98b6 (127.0.203.195:43245)
I20260619 12:00:04.761157 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 5772
I20260619 12:00:04.761245 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:04.761452 815 external_mini_cluster.cc:1428] Setting key c756456f783eb48087a4b1069df6fa38
I20260619 12:00:04.762580 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:00:04.765300 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.764049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.0.203.254:36649)
0619 12:00:04.764137 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.764139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.764296 (+ 157us) server_negotiation.cc:407] Connection header received
0619 12:00:04.764362 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.764362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.764367 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.764418 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.765074 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.765075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.765076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.765079 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.765099 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.765103 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.765156 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.765231 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.765245 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:00:04.767104 5449 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:04.768898 5449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:04.773324 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.771404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:33607 (local address 127.0.0.1:53742)
0619 12:00:04.771672 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.771719 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.771787 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.772076 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.772148 (+ 72us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.772158 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.772712 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.772713 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.772734 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.772760 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.773071 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.773078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.773082 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.773200 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.773217 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
I20260619 12:00:04.773362 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.771610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.0.203.194:33607)
0619 12:00:04.771903 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.771908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.771917 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:04.771943 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.771946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.771955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.772061 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.772867 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.772878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.772878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.772884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.772930 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.772961 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.773035 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.773217 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.773240 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":165,"threads_started":1}
I20260619 12:00:04.774204 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.771434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40835 (local address 127.0.0.1:44200)
0619 12:00:04.771987 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.772001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.772054 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.772765 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.772766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.772770 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.773726 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.773727 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.773734 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.773747 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.774056 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.774057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.774059 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.774104 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.774118 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":397,"thread_start_us":93,"threads_started":1}
I20260619 12:00:04.774196 5703 tablet_service.cc:1467] Processing CreateTablet for tablet 72e174b020a7422ba67dc045d42165d2 (DEFAULT_TABLE table=test-table [id=a2d46e13bb6c4c039d906f97546fa521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:04.774441 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.771598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.0.203.193:40835)
0619 12:00:04.772552 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.772557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.772569 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:04.772600 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.772602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.772611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.772727 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.773862 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.773877 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.773877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.773884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.773923 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.773961 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.774050 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.774330 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.774352 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":133,"threads_started":1}
I20260619 12:00:04.774606 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.771434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:43245 (local address 127.0.0.1:39908)
0619 12:00:04.771987 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.772001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.772054 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.773529 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.773529 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.773531 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.773952 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.773952 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.773961 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.773976 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.774487 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.774487 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.774488 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.774531 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.774543 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":120,"threads_started":1}
I20260619 12:00:04.774669 5703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e174b020a7422ba67dc045d42165d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:04.774864 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.771702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.203.195:43245)
0619 12:00:04.773040 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.773045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.773055 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:04.773082 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.773084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.773094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.773171 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.774156 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.774159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.774160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.774163 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.774193 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.774205 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.774265 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.774771 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.774788 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":78,"threads_started":1}
I20260619 12:00:04.775583 5834 tablet_service.cc:1467] Processing CreateTablet for tablet 72e174b020a7422ba67dc045d42165d2 (DEFAULT_TABLE table=test-table [id=a2d46e13bb6c4c039d906f97546fa521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:04.775943 5834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e174b020a7422ba67dc045d42165d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:04.778093 5914 tablet_bootstrap.cc:492] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Bootstrap starting.
I20260619 12:00:04.778942 5915 tablet_bootstrap.cc:492] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Bootstrap starting.
I20260619 12:00:04.778908 5572 tablet_service.cc:1467] Processing CreateTablet for tablet 72e174b020a7422ba67dc045d42165d2 (DEFAULT_TABLE table=test-table [id=a2d46e13bb6c4c039d906f97546fa521]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:04.779088 5914 tablet_bootstrap.cc:654] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:04.779356 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e174b020a7422ba67dc045d42165d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:04.779485 5914 log.cc:824] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:04.779714 5915 tablet_bootstrap.cc:654] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:04.780066 5915 log.cc:824] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:04.780495 5914 tablet_bootstrap.cc:492] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: No bootstrap required, opened a new log
I20260619 12:00:04.780578 5914 ts_tablet_manager.cc:1397] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:00:04.780886 5915 tablet_bootstrap.cc:492] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: No bootstrap required, opened a new log
I20260619 12:00:04.780970 5915 ts_tablet_manager.cc:1397] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:00:04.782760 5915 raft_consensus.cc:348] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.782927 5915 raft_consensus.cc:374] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:04.782971 5915 raft_consensus.cc:729] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7fb5293a5044b28931cc4a8b8b98b6, State: Initialized, Role: FOLLOWER
I20260619 12:00:04.783115 5915 consensus_queue.cc:260] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.783437 5915 ts_tablet_manager.cc:1428] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 12:00:04.783568 5900 heartbeater.cc:503] Master 127.0.203.254:36649 was elected leader, sending a full tablet report...
I20260619 12:00:04.783541 5918 tablet_bootstrap.cc:492] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148: Bootstrap starting.
I20260619 12:00:04.784099 5914 raft_consensus.cc:348] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.784291 5914 raft_consensus.cc:374] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:04.784329 5914 raft_consensus.cc:729] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3193a4b597e04393bc1af4448de2c694, State: Initialized, Role: FOLLOWER
I20260619 12:00:04.784385 5918 tablet_bootstrap.cc:654] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:04.784497 5914 consensus_queue.cc:260] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.784765 5918 log.cc:824] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:04.784863 5914 ts_tablet_manager.cc:1428] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 12:00:04.785045 5769 heartbeater.cc:503] Master 127.0.203.254:36649 was elected leader, sending a full tablet report...
I20260619 12:00:04.785635 5918 tablet_bootstrap.cc:492] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148: No bootstrap required, opened a new log
I20260619 12:00:04.785727 5918 ts_tablet_manager.cc:1397] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 12:00:04.788241 5918 raft_consensus.cc:348] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.788373 5918 raft_consensus.cc:374] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:04.788396 5918 raft_consensus.cc:729] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7609da890d864f14a7df0b542f1ed148, State: Initialized, Role: FOLLOWER
I20260619 12:00:04.788527 5918 consensus_queue.cc:260] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.788846 5918 ts_tablet_manager.cc:1428] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 12:00:04.789095 5638 heartbeater.cc:503] Master 127.0.203.254:36649 was elected leader, sending a full tablet report...
I20260619 12:00:04.874739 5922 raft_consensus.cc:482] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:04.874918 5922 raft_consensus.cc:504] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.875344 5922 leader_election.cc:288] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3193a4b597e04393bc1af4448de2c694 (127.0.203.194:33607), ab7fb5293a5044b28931cc4a8b8b98b6 (127.0.203.195:43245)
I20260619 12:00:04.877014 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.875592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:43245 (local address 127.0.203.193:40459)
0619 12:00:04.875692 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.875708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.875760 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.876001 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.876002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.876005 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.876544 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.876545 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.876554 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.876573 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.876816 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.876817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.876818 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.876900 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.876918 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 12:00:04.877063 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.875682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:40459 (local address 127.0.203.195:43245)
0619 12:00:04.875755 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.875757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.875772 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:00:04.875906 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.875911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.875920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.875995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.876699 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.876701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.876701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.876707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.876732 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.876740 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.876818 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.876919 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.876944 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 12:00:04.877276 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.875773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:42441 (local address 127.0.203.194:33607)
0619 12:00:04.875870 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.875873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.876011 (+ 138us) server_negotiation.cc:407] Connection header received
0619 12:00:04.876094 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.876095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.876103 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.876191 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.876905 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.876908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.876908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.876914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.876935 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.876940 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.877009 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.877158 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.877182 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:00:04.877270 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.875645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:33607 (local address 127.0.203.193:42441)
0619 12:00:04.875911 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.875923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:04.876024 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:04.876197 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:04.876198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:04.876201 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:04.876770 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:04.876771 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:04.876780 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:04.876795 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:04.877036 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:04.877037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:04.877038 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 12:00:04.877142 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 12:00:04.877157 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":158,"threads_started":1}
I20260619 12:00:04.877589 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e174b020a7422ba67dc045d42165d2" candidate_uuid: "7609da890d864f14a7df0b542f1ed148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" is_pre_election: true
I20260619 12:00:04.877615 5723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e174b020a7422ba67dc045d42165d2" candidate_uuid: "7609da890d864f14a7df0b542f1ed148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3193a4b597e04393bc1af4448de2c694" is_pre_election: true
I20260619 12:00:04.877804 5854 raft_consensus.cc:2454] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7609da890d864f14a7df0b542f1ed148 in term 0.
I20260619 12:00:04.877822 5723 raft_consensus.cc:2454] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7609da890d864f14a7df0b542f1ed148 in term 0.
I20260619 12:00:04.878119 5527 leader_election.cc:302] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3193a4b597e04393bc1af4448de2c694, 7609da890d864f14a7df0b542f1ed148; no voters:
I20260619 12:00:04.878297 5922 raft_consensus.cc:2781] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:04.878371 5922 raft_consensus.cc:482] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:04.878389 5922 raft_consensus.cc:3037] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:04.879230 5922 raft_consensus.cc:504] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.879400 5922 leader_election.cc:288] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [CANDIDATE]: Term 1 election: Requested vote from peers 3193a4b597e04393bc1af4448de2c694 (127.0.203.194:33607), ab7fb5293a5044b28931cc4a8b8b98b6 (127.0.203.195:43245)
I20260619 12:00:04.879565 5723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e174b020a7422ba67dc045d42165d2" candidate_uuid: "7609da890d864f14a7df0b542f1ed148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3193a4b597e04393bc1af4448de2c694"
W20260619 12:00:04.879673 5770 tablet.cc:2369] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:00:04.879699 5723 raft_consensus.cc:3037] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:04.879750 5854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e174b020a7422ba67dc045d42165d2" candidate_uuid: "7609da890d864f14a7df0b542f1ed148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6"
I20260619 12:00:04.879817 5854 raft_consensus.cc:3037] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:04.880951 5854 raft_consensus.cc:2454] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7609da890d864f14a7df0b542f1ed148 in term 1.
I20260619 12:00:04.880951 5723 raft_consensus.cc:2454] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7609da890d864f14a7df0b542f1ed148 in term 1.
I20260619 12:00:04.881194 5525 leader_election.cc:302] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7609da890d864f14a7df0b542f1ed148, ab7fb5293a5044b28931cc4a8b8b98b6; no voters:
I20260619 12:00:04.881350 5922 raft_consensus.cc:2781] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:04.881567 5922 raft_consensus.cc:686] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [term 1 LEADER]: Becoming Leader. State: Replica: 7609da890d864f14a7df0b542f1ed148, State: Running, Role: LEADER
I20260619 12:00:04.881692 5922 consensus_queue.cc:237] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } }
I20260619 12:00:04.882714 5449 catalog_manager.cc:5495] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7609da890d864f14a7df0b542f1ed148 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "7609da890d864f14a7df0b542f1ed148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7609da890d864f14a7df0b542f1ed148" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 } health_report { overall_health: UNKNOWN } } }
I20260619 12:00:04.891774 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.890665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.0.203.193:40835)
0619 12:00:04.890742 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:04.890744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:04.890753 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:04.890790 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:04.890791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:04.890803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:04.890870 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:04.891514 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:04.891516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:04.891516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:04.891521 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:04.891542 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:04.891546 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:04.891616 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:04.891675 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 12:00:04.891690 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:00:04.894254 5723 raft_consensus.cc:1261] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 1 FOLLOWER]: Refusing update from remote peer 7609da890d864f14a7df0b542f1ed148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:04.894227 5854 raft_consensus.cc:1261] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 1 FOLLOWER]: Refusing update from remote peer 7609da890d864f14a7df0b542f1ed148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:04.894688 5924 consensus_queue.cc:1035] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7fb5293a5044b28931cc4a8b8b98b6" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:04.894907 5922 consensus_queue.cc:1035] T 72e174b020a7422ba67dc045d42165d2 P 7609da890d864f14a7df0b542f1ed148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3193a4b597e04393bc1af4448de2c694" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:04.898655 5928 mvcc.cc:205] Tried to move back new op lower bound from 7298541178442412032 to 7298541178396180480. Current Snapshot: MvccSnapshot[applied={T|T < 7298541178442412032}]
I20260619 12:00:04.899782 5930 mvcc.cc:205] Tried to move back new op lower bound from 7298541178442412032 to 7298541178396180480. Current Snapshot: MvccSnapshot[applied={T|T < 7298541178442412032 or (T in {7298541178442412032})}]
I20260619 12:00:04.901139 5449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:04.901250 5449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:04.902285 5449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ed4184a04fb4e3f9e628d9e0a225898: Sending DeleteTablet for 3 replicas of tablet 72e174b020a7422ba67dc045d42165d2
I20260619 12:00:04.902709 5703 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e174b020a7422ba67dc045d42165d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:04 UTC) from {username='slave'} at 127.0.0.1:53742
I20260619 12:00:04.902709 5834 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e174b020a7422ba67dc045d42165d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:04 UTC) from {username='slave'} at 127.0.0.1:39908
I20260619 12:00:04.902997 5572 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e174b020a7422ba67dc045d42165d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:04 UTC) from {username='slave'} at 127.0.0.1:44200
I20260619 12:00:04.903040 5937 tablet_replica.cc:330] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: stopping tablet replica
I20260619 12:00:04.903138 5937 raft_consensus.cc:2229] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:04.903246 5937 raft_consensus.cc:2258] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:04.903244 5938 tablet_replica.cc:330] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: stopping tablet replica
I20260619 12:00:04.903373 5938 raft_consensus.cc:2229] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:04.903443 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 5510
I20260619 12:00:04.903564 5938 raft_consensus.cc:2258] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:04.904507 5937 ts_tablet_manager.cc:1905] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:04.905762 5938 ts_tablet_manager.cc:1905] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:04.906642 5937 ts_tablet_manager.cc:1918] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:04.906726 5937 log.cc:1196] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/72e174b020a7422ba67dc045d42165d2
I20260619 12:00:04.907011 5937 ts_tablet_manager.cc:1939] T 72e174b020a7422ba67dc045d42165d2 P 3193a4b597e04393bc1af4448de2c694: Deleting consensus metadata
I20260619 12:00:04.907387 5938 ts_tablet_manager.cc:1918] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:04.907473 5938 log.cc:1196] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/72e174b020a7422ba67dc045d42165d2
I20260619 12:00:04.907514 5436 catalog_manager.cc:4841] TS 3193a4b597e04393bc1af4448de2c694 (127.0.203.194:33607): tablet 72e174b020a7422ba67dc045d42165d2 (table test-table [id=a2d46e13bb6c4c039d906f97546fa521]) successfully deleted
I20260619 12:00:04.907742 5938 ts_tablet_manager.cc:1939] T 72e174b020a7422ba67dc045d42165d2 P ab7fb5293a5044b28931cc4a8b8b98b6: Deleting consensus metadata
I20260619 12:00:04.908238 5434 catalog_manager.cc:4841] TS ab7fb5293a5044b28931cc4a8b8b98b6 (127.0.203.195:43245): tablet 72e174b020a7422ba67dc045d42165d2 (table test-table [id=a2d46e13bb6c4c039d906f97546fa521]) successfully deleted
W20260619 12:00:04.910624 5437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:40835 (error 108)
I20260619 12:00:04.910948 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 5641
I20260619 12:00:04.911178 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:04.911051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40835 (local address 127.0.0.1:44204)
0619 12:00:04.911103 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:04.911140 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 12:00:04.911856 5437 catalog_manager.cc:4584] TS 7609da890d864f14a7df0b542f1ed148 (127.0.203.193:40835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72e174b020a7422ba67dc045d42165d2: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40835: connect: Connection refused (error 111)
I20260619 12:00:04.916991 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 5772
I20260619 12:00:04.923007 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 5419
2026-06-19T12:00:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (744 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 12:00:04.944906 815 mini_oidc.cc:152] Starting JWKS server
I20260619 12:00:04.946089 815 webserver.cc:460] Webserver started at https://127.0.0.1:35501/ using document root <none> and password file <none>
I20260619 12:00:04.946170 815 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:00:04.946374 815 webserver.cc:460] Webserver started at http://127.0.0.1:35121/ using document root <none> and password file <none>
2026-06-19T12:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:04Z Disabled control of system clock
I20260619 12:00:04.955796 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34355
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:40269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:34355
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/testchainca.cert with env {}
W20260619 12:00:05.053541 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:05.053812 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:05.053839 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:05.055179 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:05.055234 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:05.055248 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:05.055261 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:05.055274 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:00:05.055290 5946 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:00:05.055302 5946 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35501/jwks.json
I20260619 12:00:05.057041 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:34355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:05.057343 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:05.057651 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:05.063304 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:05.063310 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:05.063304 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:05.063376 5946 server_base.cc:1034] running on GCE node
I20260619 12:00:05.064098 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:05.064479 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:05.065641 5946 hybrid_clock.cc:648] HybridClock initialized: now 1781870405065632 us; error 71 us; skew 500 ppm
I20260619 12:00:05.067178 5946 webserver.cc:460] Webserver started at http://127.0.203.254:35913/ using document root <none> and password file <none>
I20260619 12:00:05.067466 5946 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:05.067520 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:05.067656 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:05.068910 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "40c54011b04140928da5fc478275ffe6"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "b45b26b1c7f40bdd03e08a1cfb108594"
server_key_iv: "2b8714e274e02cb5689a34029e892cfd"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.069294 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "40c54011b04140928da5fc478275ffe6"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "b45b26b1c7f40bdd03e08a1cfb108594"
server_key_iv: "2b8714e274e02cb5689a34029e892cfd"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.070919 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:05.072046 5960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:05.072342 5946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:05.072463 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "40c54011b04140928da5fc478275ffe6"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "b45b26b1c7f40bdd03e08a1cfb108594"
server_key_iv: "2b8714e274e02cb5689a34029e892cfd"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.072582 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:05.085978 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:05.086344 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:05.086510 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:05.091017 5946 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:34355
I20260619 12:00:05.091077 6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:34355 every 8 connection(s)
I20260619 12:00:05.091467 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:05.091809 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 5946
I20260619 12:00:05.091944 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:05.092214 815 external_mini_cluster.cc:1428] Setting key 9e710c9bedde21f729caa036d13aafbe
I20260619 12:00:05.093019 6013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:05.096886 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6: Bootstrap starting.
I20260619 12:00:05.097738 6013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:05.098096 6013 log.cc:824] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:05.098881 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6: No bootstrap required, opened a new log
I20260619 12:00:05.099141 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.092925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.0.203.254:34355)
0619 12:00:05.093158 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.093168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.093211 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:00:05.093686 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.093693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.093758 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.093899 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:05.094219 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.095262 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.096771 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.097365 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.097502 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.098277 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:05.098287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:05.098287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:05.098497 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:05.098537 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:05.098550 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:05.098646 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:05.098814 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:00:05.098908 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260619 12:00:05.100453 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:00:05.101567 6013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c54011b04140928da5fc478275ffe6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34355 } }
I20260619 12:00:05.101799 6013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:05.101842 6013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c54011b04140928da5fc478275ffe6, State: Initialized, Role: FOLLOWER
I20260619 12:00:05.101967 6013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c54011b04140928da5fc478275ffe6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34355 } }
I20260619 12:00:05.102061 6013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:05.102088 6013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:05.102125 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:05.102739 6013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c54011b04140928da5fc478275ffe6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34355 } }
I20260619 12:00:05.102833 6013 leader_election.cc:302] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c54011b04140928da5fc478275ffe6; no voters:
I20260619 12:00:05.103018 6013 leader_election.cc:288] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:05.103153 6018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:05.103382 6013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:05.103421 6018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [term 1 LEADER]: Becoming Leader. State: Replica: 40c54011b04140928da5fc478275ffe6, State: Running, Role: LEADER
I20260619 12:00:05.103607 6018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c54011b04140928da5fc478275ffe6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34355 } }
I20260619 12:00:05.104514 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c54011b04140928da5fc478275ffe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c54011b04140928da5fc478275ffe6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34355 } } }
I20260619 12:00:05.104628 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.105101 6027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:05.105516 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c54011b04140928da5fc478275ffe6. Latest consensus state: current_term: 1 leader_uuid: "40c54011b04140928da5fc478275ffe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c54011b04140928da5fc478275ffe6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34355 } } }
I20260619 12:00:05.105618 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.105793 6027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:05.107640 6027 catalog_manager.cc:1347] Generated new cluster ID: 1e4b500ba6e2445692fd861c96ddcbae
I20260619 12:00:05.107712 6027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:05.119388 6027 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:05.120276 6027 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:05.132680 6027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c54011b04140928da5fc478275ffe6: Generated new TSK 0
I20260619 12:00:05.133138 6027 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 12:00:05.160503 6043 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 12:00:05.160650 6043 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260619 12:00:05.161461 6042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:05.143115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:34355 (local address 127.0.0.1:57194)
0619 12:00:05.143465 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:05.143477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:05.143558 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:05.143878 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:05.143879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:05.143882 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:00:05.143968 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.143970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.145088 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.145089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.145811 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.145813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.146337 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.146338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.146425 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.146505 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:00:05.160884 (+ 14379us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 12:00:05.161037 (+ 153us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.254:34355: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":197,"threads_started":1}
I20260619 12:00:05.162134 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.143294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.0.203.254:34355)
0619 12:00:05.143376 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.143379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.143555 (+ 176us) server_negotiation.cc:407] Connection header received
0619 12:00:05.143678 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.143679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.143696 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.143816 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:00:05.144066 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.144966 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.145924 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.146240 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.146318 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.146640 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:00:05.160629 (+ 13989us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 12:00:05.161773 (+ 1144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 12:00:05.162245 6016 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.203.254:34355: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 12:00:05.162596 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 5946
2026-06-19T12:00:05Z chronyd exiting
I20260619 12:00:05.201473 815 test_util.cc:175] -----------------------------------------------
I20260619 12:00:05.201608 815 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781870380032316-815-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (257 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 12:00:05.202615 815 mini_oidc.cc:152] Starting JWKS server
I20260619 12:00:05.203434 815 webserver.cc:460] Webserver started at https://127.0.0.1:39773/ using document root <none> and password file <none>
I20260619 12:00:05.203485 815 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:00:05.203652 815 webserver.cc:460] Webserver started at http://127.0.0.1:46849/ using document root <none> and password file <none>
2026-06-19T12:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:05Z Disabled control of system clock
I20260619 12:00:05.213394 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:45989
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:45989
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39773/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/testchainca.cert with env {}
W20260619 12:00:05.304544 6049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:05.304844 6049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:05.304880 6049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:05.306114 6049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:05.306180 6049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:05.306201 6049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:05.306231 6049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:05.306262 6049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:00:05.306286 6049 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:00:05.306308 6049 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39773/jwks.json
I20260619 12:00:05.307874 6049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:45989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39773/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:05.308171 6049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:05.308423 6049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:05.314038 6057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:05.314060 6054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:05.314060 6055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:05.314512 6049 server_base.cc:1034] running on GCE node
I20260619 12:00:05.314790 6049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:05.315138 6049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:05.316318 6049 hybrid_clock.cc:648] HybridClock initialized: now 1781870405316281 us; error 70 us; skew 500 ppm
I20260619 12:00:05.317939 6049 webserver.cc:460] Webserver started at http://127.0.203.254:46119/ using document root <none> and password file <none>
I20260619 12:00:05.318223 6049 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:05.318280 6049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:05.318408 6049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:05.319655 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "af2f62b35b8349b19a7e2926d0c64c0b"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "3e64bc6f809b1bf5f5a8795f3f964a89"
server_key_iv: "4e2c103522ea838db59331da64040762"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.320036 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "af2f62b35b8349b19a7e2926d0c64c0b"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "3e64bc6f809b1bf5f5a8795f3f964a89"
server_key_iv: "4e2c103522ea838db59331da64040762"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.321599 6049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:05.322571 6063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:05.322835 6049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:05.322954 6049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "af2f62b35b8349b19a7e2926d0c64c0b"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "3e64bc6f809b1bf5f5a8795f3f964a89"
server_key_iv: "4e2c103522ea838db59331da64040762"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.323035 6049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:05.335745 6049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:05.336109 6049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:05.336272 6049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:05.340541 6049 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:45989
I20260619 12:00:05.340660 6115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:45989 every 8 connection(s)
I20260619 12:00:05.341041 6049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:05.342280 6116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:05.345036 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b: Bootstrap starting.
I20260619 12:00:05.345773 6116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:05.346119 6116 log.cc:824] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:05.347127 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b: No bootstrap required, opened a new log
I20260619 12:00:05.349004 6116 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45989 } }
I20260619 12:00:05.349161 6116 raft_consensus.cc:374] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:05.349185 6116 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2f62b35b8349b19a7e2926d0c64c0b, State: Initialized, Role: FOLLOWER
I20260619 12:00:05.349367 6116 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45989 } }
I20260619 12:00:05.349474 6116 raft_consensus.cc:388] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:05.349510 6116 raft_consensus.cc:482] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:05.349560 6116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:05.349562 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6049
I20260619 12:00:05.349654 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:05.349862 815 external_mini_cluster.cc:1428] Setting key 144e9645aab131dfdf82537515bc60a3
I20260619 12:00:05.350308 6116 raft_consensus.cc:504] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45989 } }
I20260619 12:00:05.350448 6116 leader_election.cc:302] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af2f62b35b8349b19a7e2926d0c64c0b; no voters:
I20260619 12:00:05.350657 6116 leader_election.cc:288] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:05.350754 6121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:05.350888 6121 raft_consensus.cc:686] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [term 1 LEADER]: Becoming Leader. State: Replica: af2f62b35b8349b19a7e2926d0c64c0b, State: Running, Role: LEADER
I20260619 12:00:05.351025 6116 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:05.351035 6121 consensus_queue.cc:237] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45989 } }
I20260619 12:00:05.351707 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader af2f62b35b8349b19a7e2926d0c64c0b. Latest consensus state: current_term: 1 leader_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45989 } } }
I20260619 12:00:05.351821 6123 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2f62b35b8349b19a7e2926d0c64c0b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45989 } } }
I20260619 12:00:05.351948 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.352015 6123 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.352339 6130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:05.352977 6130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:05.355146 6130 catalog_manager.cc:1347] Generated new cluster ID: b35a1da162c34a7fac01910c1713a248
I20260619 12:00:05.355221 6130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:05.356923 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.350506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.0.203.254:45989)
0619 12:00:05.350819 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.350832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.350881 (+ 49us) server_negotiation.cc:407] Connection header received
0619 12:00:05.351055 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.351063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.351154 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.351321 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:05.351656 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.353207 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.354368 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.354993 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.355467 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.355970 (+ 503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:05.355974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:05.355975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:05.356181 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:05.356221 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:05.356236 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:05.356537 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:05.356569 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 12:00:05.356679 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
I20260619 12:00:05.357915 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:00:05.365531 6130 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:05.366523 6130 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:05.371349 6130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af2f62b35b8349b19a7e2926d0c64c0b: Generated new TSK 0
I20260619 12:00:05.371589 6130 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 12:00:05.393618 6146 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 12:00:05.393729 6146 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 12:00:05.394172 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.378637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.0.203.254:45989)
0619 12:00:05.378697 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.378699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.378708 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:05.378768 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.378769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.378782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.378870 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:00:05.378978 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.379695 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.380853 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.381123 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.381177 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.381432 (+ 255us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:00:05.393900 (+ 12468us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 12:00:05.394077 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46538: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 12:00:05.394235 6145 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:05.378510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:45989 (local address 127.0.0.1:46538)
0619 12:00:05.378621 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:05.378631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:05.378693 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:05.378871 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:05.378872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:05.378873 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:00:05.378926 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.378928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.379834 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.379835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.380749 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.380752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.381249 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.381250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.381309 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.381314 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:00:05.394033 (+ 12719us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 12:00:05.394140 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.254:45989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":69,"threads_started":1}
W20260619 12:00:05.394240 6120 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46538: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39773/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 12:00:05.394572 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6049
2026-06-19T12:00:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 12:00:05.434803 815 mini_oidc.cc:152] Starting JWKS server
I20260619 12:00:05.435552 815 webserver.cc:460] Webserver started at https://127.0.0.1:36753/ using document root <none> and password file <none>
I20260619 12:00:05.435599 815 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:00:05.435763 815 webserver.cc:460] Webserver started at http://127.0.0.1:38177/ using document root <none> and password file <none>
2026-06-19T12:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:05Z Disabled control of system clock
I20260619 12:00:05.445779 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:43119
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:41787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:43119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36753/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 12:00:05.541878 6152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:05.542188 6152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:05.542229 6152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:05.543557 6152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:05.543634 6152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:05.543658 6152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:05.543682 6152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:05.543716 6152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:00:05.543744 6152 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:00:05.543776 6152 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36753/jwks.json
I20260619 12:00:05.545367 6152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:43119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:43119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36753/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:05.545723 6152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:05.546084 6152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:05.552062 6157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:05.552136 6160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:05.552227 6152 server_base.cc:1034] running on GCE node
W20260619 12:00:05.552084 6158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:05.552835 6152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:05.553185 6152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:05.554373 6152 hybrid_clock.cc:648] HybridClock initialized: now 1781870405554354 us; error 48 us; skew 500 ppm
I20260619 12:00:05.555922 6152 webserver.cc:460] Webserver started at http://127.0.203.254:44253/ using document root <none> and password file <none>
I20260619 12:00:05.556217 6152 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:05.556299 6152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:05.556499 6152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:05.557790 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "d3adeed091d545698f3f5d87509198da"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "62cb2965c970b03d19f277a804cc8d2b"
server_key_iv: "87d59d51fa9d6e1fa41bebae448941ab"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.558188 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "d3adeed091d545698f3f5d87509198da"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "62cb2965c970b03d19f277a804cc8d2b"
server_key_iv: "87d59d51fa9d6e1fa41bebae448941ab"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.559762 6152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:05.560806 6166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:05.561103 6152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:05.561215 6152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "d3adeed091d545698f3f5d87509198da"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "62cb2965c970b03d19f277a804cc8d2b"
server_key_iv: "87d59d51fa9d6e1fa41bebae448941ab"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.561300 6152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:05.590234 6152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:05.590719 6152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:05.590903 6152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:05.596724 6152 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:43119
I20260619 12:00:05.596791 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:43119 every 8 connection(s)
I20260619 12:00:05.597247 6152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:05.598140 6219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:05.601228 6219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da: Bootstrap starting.
I20260619 12:00:05.601824 6219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:05.602077 6219 log.cc:824] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:05.602397 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6152
I20260619 12:00:05.602494 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:05.602759 815 external_mini_cluster.cc:1428] Setting key 48e1034fe35a9a1733d85d822ee6a701
I20260619 12:00:05.603103 6219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da: No bootstrap required, opened a new log
I20260619 12:00:05.605135 6219 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adeed091d545698f3f5d87509198da" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43119 } }
I20260619 12:00:05.605302 6219 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:05.605329 6219 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3adeed091d545698f3f5d87509198da, State: Initialized, Role: FOLLOWER
I20260619 12:00:05.605472 6219 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adeed091d545698f3f5d87509198da" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43119 } }
I20260619 12:00:05.605557 6219 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:05.605607 6219 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:05.605712 6219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:05.606353 6219 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adeed091d545698f3f5d87509198da" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43119 } }
I20260619 12:00:05.606489 6219 leader_election.cc:302] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3adeed091d545698f3f5d87509198da; no voters:
I20260619 12:00:05.606717 6219 leader_election.cc:288] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:05.606763 6224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:05.607014 6224 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [term 1 LEADER]: Becoming Leader. State: Replica: d3adeed091d545698f3f5d87509198da, State: Running, Role: LEADER
I20260619 12:00:05.607136 6224 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adeed091d545698f3f5d87509198da" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43119 } }
I20260619 12:00:05.607177 6219 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:05.607540 6224 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3adeed091d545698f3f5d87509198da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adeed091d545698f3f5d87509198da" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43119 } } }
I20260619 12:00:05.607736 6224 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.607942 6224 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3adeed091d545698f3f5d87509198da. Latest consensus state: current_term: 1 leader_uuid: "d3adeed091d545698f3f5d87509198da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3adeed091d545698f3f5d87509198da" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43119 } } }
I20260619 12:00:05.608080 6224 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.608532 6231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:05.609275 6231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:05.611263 6231 catalog_manager.cc:1347] Generated new cluster ID: cf290dc6d3f54d8b956eed2e4716f76c
I20260619 12:00:05.611330 6231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:05.612434 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.603349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.0.203.254:43119)
0619 12:00:05.603616 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.603628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.603667 (+ 39us) server_negotiation.cc:407] Connection header received
0619 12:00:05.605148 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.605158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.605377 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.605671 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:05.606175 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.608625 (+ 2450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.609855 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.610499 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.610647 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.611513 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:05.611518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:05.611519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:05.611770 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:05.611812 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:05.611827 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:05.611911 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:05.612007 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 12:00:05.612223 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260619 12:00:05.613823 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:00:05.622457 6231 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:05.623207 6231 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:05.628353 6231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3adeed091d545698f3f5d87509198da: Generated new TSK 0
I20260619 12:00:05.628652 6231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:00:05.643934 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.639533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.0.203.254:43119)
0619 12:00:05.639585 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.639588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.639806 (+ 218us) server_negotiation.cc:407] Connection header received
0619 12:00:05.639860 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.639862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.639876 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.639972 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:00:05.640197 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.640847 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.642164 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.642357 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.642428 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.642702 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:00:05.643670 (+ 968us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 12:00:05.643837 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 12:00:05.644008 6223 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 12:00:05.644037 6248 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:05.639381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:43119 (local address 127.0.0.1:54990)
0619 12:00:05.639625 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:05.639645 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:05.639765 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:05.639991 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:05.639992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:05.639994 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:00:05.640082 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.640085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.640984 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.640985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.642022 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.642025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.642463 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.642464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.642530 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.642538 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:00:05.643799 (+ 1261us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 12:00:05.643915 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.254:43119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36753/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":130,"threads_started":1}
I20260619 12:00:05.644487 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6152
E20260619 12:00:05.657107 6249 webserver.cc:533] Webserver: sslize failed:
E20260619 12:00:05.657225 6249 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T12:00:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (249 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T12:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:05Z Disabled control of system clock
I20260619 12:00:05.693383 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:38073
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:38073
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/testchainca.cert with env {}
W20260619 12:00:05.778520 6253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:05.778764 6253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:05.778785 6253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:05.779959 6253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:05.780009 6253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:05.780022 6253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:05.780035 6253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:05.780048 6253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:00:05.780061 6253 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:00:05.780104 6253 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 12:00:05.781615 6253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:38073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:05.781891 6253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:05.782119 6253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:05.787942 6258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:05.787997 6259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:05.788012 6253 server_base.cc:1034] running on GCE node
W20260619 12:00:05.787942 6261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:05.788594 6253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:05.788995 6253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:05.790179 6253 hybrid_clock.cc:648] HybridClock initialized: now 1781870405790146 us; error 61 us; skew 500 ppm
I20260619 12:00:05.791599 6253 webserver.cc:460] Webserver started at http://127.0.203.254:38591/ using document root <none> and password file <none>
I20260619 12:00:05.791865 6253 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:05.791940 6253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:05.792091 6253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:05.793273 6253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "5bedeeead27a06ee4cd968343b9b8fa6"
server_key_iv: "5324823700e5a2408f345241d392820a"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.793648 6253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "5bedeeead27a06ee4cd968343b9b8fa6"
server_key_iv: "5324823700e5a2408f345241d392820a"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.795123 6253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:05.796190 6267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:05.796535 6253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:05.796715 6253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9"
format_stamp: "Formatted at 2026-06-19 12:00:05 on dist-test-slave-fbq8"
server_key: "5bedeeead27a06ee4cd968343b9b8fa6"
server_key_iv: "5324823700e5a2408f345241d392820a"
server_key_version: "encryptionkey@0"
I20260619 12:00:05.796882 6253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:05.805372 6253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:05.805712 6253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:05.805882 6253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:05.810436 6253 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:38073
I20260619 12:00:05.810506 6319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:38073 every 8 connection(s)
I20260619 12:00:05.810884 6253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:05.812201 6320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:05.815034 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9: Bootstrap starting.
I20260619 12:00:05.815716 6320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:05.816004 6320 log.cc:824] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:05.816926 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9: No bootstrap required, opened a new log
I20260619 12:00:05.818734 6320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38073 } }
I20260619 12:00:05.818975 6320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:05.819025 6320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f90a2cdc8774425b5b9d72bc0e16ab9, State: Initialized, Role: FOLLOWER
I20260619 12:00:05.819062 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6253
I20260619 12:00:05.819187 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:05.819211 6320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38073 } }
I20260619 12:00:05.819315 6320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:05.819365 6320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:05.819423 6320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:05.819435 815 external_mini_cluster.cc:1428] Setting key 71c7c4c0f8502cc466f3421e11b1a58c
I20260619 12:00:05.820298 6320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38073 } }
I20260619 12:00:05.820443 6320 leader_election.cc:302] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f90a2cdc8774425b5b9d72bc0e16ab9; no voters:
I20260619 12:00:05.820717 6320 leader_election.cc:288] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:05.820750 6325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:05.820890 6325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 7f90a2cdc8774425b5b9d72bc0e16ab9, State: Running, Role: LEADER
I20260619 12:00:05.820991 6325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38073 } }
I20260619 12:00:05.821079 6320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:05.821393 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38073 } } }
I20260619 12:00:05.821750 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.821657 6326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f90a2cdc8774425b5b9d72bc0e16ab9. Latest consensus state: current_term: 1 leader_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90a2cdc8774425b5b9d72bc0e16ab9" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38073 } } }
I20260619 12:00:05.822054 6326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:05.822297 6334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:05.822957 6334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:05.824896 6334 catalog_manager.cc:1347] Generated new cluster ID: 27010dadcf644654bb163b7543e1b79c
I20260619 12:00:05.825008 6334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:05.826954 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.820087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.203.254:38073)
0619 12:00:05.820341 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.820351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.820387 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:00:05.820551 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.820559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.820705 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.821199 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:05.821312 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.823110 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.824254 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.824880 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.825017 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.825939 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:05.825943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:05.825944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:05.826188 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:05.826234 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:05.826250 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:05.826328 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:05.826425 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 12:00:05.826635 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":127,"threads_started":1}
I20260619 12:00:05.828115 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:00:05.845041 6334 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:05.845903 6334 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:05.858332 6334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f90a2cdc8774425b5b9d72bc0e16ab9: Generated new TSK 0
I20260619 12:00:05.858682 6334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:00:06.299242 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:05.875130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.0.203.254:38073)
0619 12:00:05.875221 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:05.875224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:05.875452 (+ 228us) server_negotiation.cc:407] Connection header received
0619 12:00:05.875505 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:05.875506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:05.875522 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:05.875632 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:00:05.875808 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.876424 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.877691 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.877905 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.877981 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.878191 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:00:06.298927 (+420736us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 12:00:06.299155 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53024: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 12:00:06.299364 6324 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53024: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 12:00:06.299470 6349 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:05.875068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38073 (local address 127.0.0.1:53024)
0619 12:00:05.875295 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:05.875309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:05.875402 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:05.875654 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:05.875655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:05.875657 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:00:05.875721 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.875724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.876537 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.876538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.877567 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:05.877570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:05.878008 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:05.878009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:05.878074 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:05.878080 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:00:06.299140 (+421060us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 12:00:06.299305 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.203.254:38073: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":129,"threads_started":1}
I20260619 12:00:06.300027 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6253
2026-06-19T12:00:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (635 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (100 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:06 dist-test-slave-fbq8 krb5kdc[6361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:06 dist-test-slave-fbq8 krb5kdc[6361](info): set up 2 sockets
Jun 19 12:00:06 dist-test-slave-fbq8 krb5kdc[6361](info): commencing operation
krb5kdc: starting...
W20260619 12:00:07.560998 815 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:07 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:00:07.696178 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34865
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:07.785143 6374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:07.785440 6374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:07.785478 6374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:07.786623 6374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:07.786696 6374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:07.786721 6374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:07.786743 6374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:07.786773 6374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:07.788322 6374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:34865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:07.788674 6374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:07.788930 6374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:07.794767 6379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:07.794795 6382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:07.794761 6380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:07.795177 6374 server_base.cc:1034] running on GCE node
I20260619 12:00:07.795495 6374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:07.795822 6374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:07.796990 6374 hybrid_clock.cc:648] HybridClock initialized: now 1781870407796967 us; error 53 us; skew 500 ppm
Jun 19 12:00:07 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:07.799979 6374 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:00:07.800542 6374 webserver.cc:460] Webserver started at http://127.0.203.254:46823/ using document root <none> and password file <none>
I20260619 12:00:07.800858 6374 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:07.800930 6374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:07.801107 6374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:07.802269 6374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "6ccea01a7c0944dcaee54e46188dc22b"
format_stamp: "Formatted at 2026-06-19 12:00:07 on dist-test-slave-fbq8"
server_key: "6163d81ba9c4f5b96cb977614fe2f2e3"
server_key_iv: "92981cc902bf938850b8481aae433b02"
server_key_version: "encryptionkey@0"
I20260619 12:00:07.802662 6374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "6ccea01a7c0944dcaee54e46188dc22b"
format_stamp: "Formatted at 2026-06-19 12:00:07 on dist-test-slave-fbq8"
server_key: "6163d81ba9c4f5b96cb977614fe2f2e3"
server_key_iv: "92981cc902bf938850b8481aae433b02"
server_key_version: "encryptionkey@0"
I20260619 12:00:07.804239 6374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:07.805238 6389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:07.805541 6374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:00:07.805650 6374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "6ccea01a7c0944dcaee54e46188dc22b"
format_stamp: "Formatted at 2026-06-19 12:00:07 on dist-test-slave-fbq8"
server_key: "6163d81ba9c4f5b96cb977614fe2f2e3"
server_key_iv: "92981cc902bf938850b8481aae433b02"
server_key_version: "encryptionkey@0"
I20260619 12:00:07.805737 6374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:07.813596 6374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:07.815935 6374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:07.816097 6374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:07.821319 6374 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:34865
I20260619 12:00:07.821727 6374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:07.821813 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:34865 every 8 connection(s)
I20260619 12:00:07.821995 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6374
I20260619 12:00:07.822118 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:07.822373 815 external_mini_cluster.cc:1428] Setting key 4b49f23183eedf9346935d4b65c8d8c9
I20260619 12:00:07.822665 6442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:07.826128 6442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b: Bootstrap starting.
I20260619 12:00:07.826840 6442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:07.827157 6442 log.cc:824] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:07.827913 6442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b: No bootstrap required, opened a new log
Jun 19 12:00:07 dist-test-slave-fbq8 krb5kdc[6361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:07.830760 6442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccea01a7c0944dcaee54e46188dc22b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34865 } }
I20260619 12:00:07.831081 6442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:07.831156 6442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccea01a7c0944dcaee54e46188dc22b, State: Initialized, Role: FOLLOWER
I20260619 12:00:07.831488 6442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccea01a7c0944dcaee54e46188dc22b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34865 } }
I20260619 12:00:07.831604 6442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:07.831657 6442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:07.831705 6442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:07.832440 6442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccea01a7c0944dcaee54e46188dc22b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34865 } }
I20260619 12:00:07.832567 6442 leader_election.cc:302] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ccea01a7c0944dcaee54e46188dc22b; no voters:
I20260619 12:00:07.832824 6442 leader_election.cc:288] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:07.832903 6447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:07.833150 6442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:07.833154 6447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [term 1 LEADER]: Becoming Leader. State: Replica: 6ccea01a7c0944dcaee54e46188dc22b, State: Running, Role: LEADER
I20260619 12:00:07.833470 6447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccea01a7c0944dcaee54e46188dc22b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34865 } }
I20260619 12:00:07.834100 6449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ccea01a7c0944dcaee54e46188dc22b. Latest consensus state: current_term: 1 leader_uuid: "6ccea01a7c0944dcaee54e46188dc22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccea01a7c0944dcaee54e46188dc22b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34865 } } }
I20260619 12:00:07.834220 6449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:07.834311 6448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ccea01a7c0944dcaee54e46188dc22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ccea01a7c0944dcaee54e46188dc22b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34865 } } }
I20260619 12:00:07.834368 6448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:07.834517 6456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:07.834595 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:07.822934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.0.203.254:34865)
0619 12:00:07.823189 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:07.823200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:07.823232 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:00:07.823723 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:07.823731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:07.823777 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:07.823932 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:07.824603 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:07.825609 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:07.826454 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:07.826898 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:07.827020 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:07.830195 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:07.830200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:07.830200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:07.830218 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:07.832112 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:07.832487 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:07.832490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:07.832490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:07.832602 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:07.833042 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:07.833044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:07.833045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:07.833437 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:07.833618 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:07.833764 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 12:00:07.833968 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":115,"threads_started":1}
I20260619 12:00:07.835537 6456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:07.837375 6456 catalog_manager.cc:1347] Generated new cluster ID: 0c3cf93ca26b4f45839b42e765e32b53
I20260619 12:00:07.837436 6456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:07.877652 6456 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:07.878449 6456 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:07.888882 6456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ccea01a7c0944dcaee54e46188dc22b: Generated new TSK 0
I20260619 12:00:07.889122 6456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:00:07.928079 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:34865
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:08.022773 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:08.023031 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:08.023067 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:08.024320 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:08.024425 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:08.024473 6468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:08.026014 6468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:34865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:08.026305 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:08.026552 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:08.032375 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:08.032433 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:08.032480 6468 server_base.cc:1034] running on GCE node
W20260619 12:00:08.032354 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:08.032967 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:08.033243 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:08.034416 6468 hybrid_clock.cc:648] HybridClock initialized: now 1781870408034373 us; error 72 us; skew 500 ppm
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.037562 6468 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:00:08.038097 6468 webserver.cc:460] Webserver started at http://127.0.203.193:34725/ using document root <none> and password file <none>
I20260619 12:00:08.038375 6468 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:08.038425 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:08.038570 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:08.039744 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "f0802756cb6548298bef5d106d13628f"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "c301a80f853545efc5db8803941d6b0c"
server_key_iv: "21599dd279925ce14b64aa4610e1feb3"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.040149 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "f0802756cb6548298bef5d106d13628f"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "c301a80f853545efc5db8803941d6b0c"
server_key_iv: "21599dd279925ce14b64aa4610e1feb3"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.041838 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:08.042837 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.043099 6468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:08.043329 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "f0802756cb6548298bef5d106d13628f"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "c301a80f853545efc5db8803941d6b0c"
server_key_iv: "21599dd279925ce14b64aa4610e1feb3"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.043447 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:08.066370 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:08.069029 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:08.069224 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:08.069567 6468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:08.069926 6468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:08.069984 6468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.070030 6468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:08.070075 6468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.078140 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:36853
I20260619 12:00:08.078289 6596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:36853 every 8 connection(s)
I20260619 12:00:08.078684 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:08.084856 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6468
I20260619 12:00:08.084960 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:08.085181 815 external_mini_cluster.cc:1428] Setting key e92b8225af1f6fc5eff1a229be374126
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:08.090036 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.079661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:48821 (local address 127.0.203.254:34865)
0619 12:00:08.079788 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.079791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.079995 (+ 204us) server_negotiation.cc:407] Connection header received
0619 12:00:08.081101 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.081102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.081115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.081217 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.081987 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.082500 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.083549 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.083759 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.083841 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.086675 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:08.086676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:08.086677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:08.086686 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:08.088383 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.088946 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.088947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.088948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.089015 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.089288 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.089289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.089289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.089442 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:08.089511 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:08.089811 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 12:00:08.089923 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 12:00:08.090231 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.079550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:34865 (local address 127.0.203.193:48821)
0619 12:00:08.079833 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:08.079885 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:08.080974 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:08.081312 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:08.081316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:08.081618 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:08.081872 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:08.081876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.082596 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.082597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:08.083455 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:08.083458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.083849 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.083850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:08.083980 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.084588 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:08.084598 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:08.086551 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:08.088526 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:08.088541 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:08.088543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:08.088863 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:08.089098 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:08.089099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:08.089099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:08.089207 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:08.089511 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:08.089515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:08.089653 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 12:00:08.089737 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:00:08.089885 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260619 12:00:08.091064 6597 heartbeater.cc:347] Connected to a master server at 127.0.203.254:34865
I20260619 12:00:08.091224 6597 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:08.091475 6597 heartbeater.cc:511] Master 127.0.203.254:34865 requested a full tablet report, sending...
I20260619 12:00:08.092173 6406 ts_manager.cc:194] Registered new tserver with Master: f0802756cb6548298bef5d106d13628f (127.0.203.193:36853)
I20260619 12:00:08.093147 6406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:48821
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:00:08.115599 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:34865
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:08.203625 6602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:08.203922 6602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:08.203967 6602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:08.205358 6602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:08.205454 6602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:08.205509 6602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:08.207237 6602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:34865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:08.207576 6602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:08.207851 6602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:08.213986 6608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:08.214031 6602 server_base.cc:1034] running on GCE node
W20260619 12:00:08.214017 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:08.213986 6610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:08.214672 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:08.214993 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:08.216166 6602 hybrid_clock.cc:648] HybridClock initialized: now 1781870408216105 us; error 105 us; skew 500 ppm
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.219904 6602 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:00:08.220501 6602 webserver.cc:460] Webserver started at http://127.0.203.194:46257/ using document root <none> and password file <none>
I20260619 12:00:08.220839 6602 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:08.220917 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:08.221100 6602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:08.222415 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "a20c49680b804647afb50f073f165938"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "bcad356b199a4920407701b1f1afd3dc"
server_key_iv: "97d018a2a7dd413d09017fddeae88ca4"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.222838 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "a20c49680b804647afb50f073f165938"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "bcad356b199a4920407701b1f1afd3dc"
server_key_iv: "97d018a2a7dd413d09017fddeae88ca4"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.225775 6602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 12:00:08.227746 6617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.228195 6602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:08.228368 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "a20c49680b804647afb50f073f165938"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "bcad356b199a4920407701b1f1afd3dc"
server_key_iv: "97d018a2a7dd413d09017fddeae88ca4"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.228480 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:08.240430 6602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:08.243171 6602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:08.243348 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:08.243637 6602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:08.243989 6602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:08.244045 6602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.244095 6602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:08.244118 6602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.252146 6602 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:39595
I20260619 12:00:08.252200 6730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:39595 every 8 connection(s)
I20260619 12:00:08.252599 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:08.261821 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6602
I20260619 12:00:08.261930 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:08.262168 815 external_mini_cluster.cc:1428] Setting key 96871f4133b0630a6a5d2b9bdb85f9f6
I20260619 12:00:08.265591 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.253685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:39273 (local address 127.0.203.254:34865)
0619 12:00:08.253782 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.253786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.254055 (+ 269us) server_negotiation.cc:407] Connection header received
0619 12:00:08.255255 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.255257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.255269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.255341 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.256346 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.256894 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.258036 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.258245 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.258310 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.261233 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:08.261235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:08.261235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:08.261244 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:08.263408 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.264156 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.264157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.264157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.264234 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.264601 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.264603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.264603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.264831 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:08.264921 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:08.265310 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 12:00:08.265438 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:00:08.265628 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.253625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:34865 (local address 127.0.203.194:39273)
0619 12:00:08.253875 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:08.253924 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:08.255114 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:08.255533 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:08.255539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:08.255908 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:08.256216 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:08.256223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.257011 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.257012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:08.257946 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:08.257949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.258382 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.258384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:08.258542 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.259296 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:08.259306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:08.261119 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:08.263517 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:08.263532 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:08.263535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:08.264050 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:08.264314 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:08.264315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:08.264315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:08.264491 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:08.264974 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:08.264980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:08.265138 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 12:00:08.265258 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 12:00:08.265428 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":145,"threads_started":1}
I20260619 12:00:08.266269 6731 heartbeater.cc:347] Connected to a master server at 127.0.203.254:34865
I20260619 12:00:08.266400 6731 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:08.266601 6731 heartbeater.cc:511] Master 127.0.203.254:34865 requested a full tablet report, sending...
I20260619 12:00:08.267189 6406 ts_manager.cc:194] Registered new tserver with Master: a20c49680b804647afb50f073f165938 (127.0.203.194:39595)
I20260619 12:00:08.267689 6406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:39273
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:00:08.293171 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:34865
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:08.381203 6736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:08.381448 6736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:08.381474 6736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:08.382642 6736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:08.382688 6736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:08.382717 6736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:08.385092 6736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:34865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:08.385326 6736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:08.385519 6736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:08.391281 6744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:08.391351 6736 server_base.cc:1034] running on GCE node
W20260619 12:00:08.391413 6741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:08.391402 6742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:08.391863 6736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:08.392206 6736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:08.393430 6736 hybrid_clock.cc:648] HybridClock initialized: now 1781870408393393 us; error 51 us; skew 500 ppm
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.397616 6736 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:00:08.398334 6736 webserver.cc:460] Webserver started at http://127.0.203.195:34431/ using document root <none> and password file <none>
I20260619 12:00:08.398701 6736 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:08.398823 6736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:08.398991 6736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:08.400187 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "de635d7d588941b5a57f82057b000e29"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "dbac56a9f77038608408a826016e8128"
server_key_iv: "770f69b7de16bdec2cd9bc28b1f159d7"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.400743 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "de635d7d588941b5a57f82057b000e29"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "dbac56a9f77038608408a826016e8128"
server_key_iv: "770f69b7de16bdec2cd9bc28b1f159d7"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.402343 6736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:08.403344 6751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.403656 6736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:08.403777 6736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "de635d7d588941b5a57f82057b000e29"
format_stamp: "Formatted at 2026-06-19 12:00:08 on dist-test-slave-fbq8"
server_key: "dbac56a9f77038608408a826016e8128"
server_key_iv: "770f69b7de16bdec2cd9bc28b1f159d7"
server_key_version: "encryptionkey@0"
I20260619 12:00:08.403878 6736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:08.425052 6736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:08.427258 6736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:08.427438 6736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:08.427716 6736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:08.428048 6736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:08.428107 6736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.428176 6736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:08.428221 6736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:08.435920 6736 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:44521
I20260619 12:00:08.435971 6864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:44521 every 8 connection(s)
I20260619 12:00:08.436335 6736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:08.439862 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6736
I20260619 12:00:08.440001 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:08.440188 815 external_mini_cluster.cc:1428] Setting key f1867c83dd5a124aae22820c2b44ab02
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:08.448119 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.437310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:41789 (local address 127.0.203.254:34865)
0619 12:00:08.437432 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.437434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.437636 (+ 202us) server_negotiation.cc:407] Connection header received
0619 12:00:08.438761 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.438762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.438775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.438866 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.439604 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.440227 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.441991 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.442226 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.442297 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.445062 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:08.445064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:08.445064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:08.445073 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:08.446597 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.447072 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.447073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.447074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.447137 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.447395 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.447395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.447396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.447526 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:08.447606 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:08.447937 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 12:00:08.448017 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 12:00:08.448331 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.437247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:34865 (local address 127.0.203.195:41789)
0619 12:00:08.437472 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:08.437523 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:08.438615 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:08.438939 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:08.438944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:08.439256 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:08.439494 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:08.439498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.440359 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.440360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:08.441850 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:08.441856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.442349 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.442350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:08.442501 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.443139 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:08.443149 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:08.444943 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:08.446693 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:08.446703 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:08.446705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:08.446990 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:08.447207 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:08.447208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:08.447208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:08.447324 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:08.447597 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:08.447603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:08.447762 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 12:00:08.447899 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 12:00:08.448085 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":134,"threads_started":1}
I20260619 12:00:08.448983 6865 heartbeater.cc:347] Connected to a master server at 127.0.203.254:34865
I20260619 12:00:08.449076 6865 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:08.449227 6865 heartbeater.cc:511] Master 127.0.203.254:34865 requested a full tablet report, sending...
I20260619 12:00:08.449623 6406 ts_manager.cc:194] Registered new tserver with Master: de635d7d588941b5a57f82057b000e29 (127.0.203.195:44521)
I20260619 12:00:08.450124 6406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:41789
I20260619 12:00:08.452216 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/test-admin.keytab.
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.470155 815 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 12:00:08.472288 6872 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:08.471275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.0.203.193:36853)
0619 12:00:08.471518 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.471524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.471534 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:08.471566 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.471568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.471586 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.471695 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.472164 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46952: BlockingRecv error: recv got EOF from 127.0.0.1:46952 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
W20260619 12:00:08.472471 815 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc)
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.473770 815 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.474869 815 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 12:00:08.484961 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.475574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.203.193:36853)
0619 12:00:08.475655 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.475657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.475684 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:00:08.475779 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.475780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.475800 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.475874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.476304 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.477212 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.477979 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.478327 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.478415 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.481857 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:08.481870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:08.481870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:08.481884 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:08.483591 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.484016 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.484019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.484020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.484086 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.484290 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.484291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.484292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.484458 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:08.484707 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:08.484756 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 12:00:08.484836 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 12:00:08.487713 6872 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:08.486867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.0.203.193:36853)
0619 12:00:08.486918 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.486919 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.487075 (+ 156us) server_negotiation.cc:407] Connection header received
0619 12:00:08.487098 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.487099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.487111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.487171 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.487658 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46962: BlockingRecv error: recv got EOF from 127.0.0.1:46962 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 12:00:08.487892 815 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc' of type FILE
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.489100 815 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.490003 815 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 12:00:08.498064 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.490610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.0.203.193:36853)
0619 12:00:08.490657 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.490659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.490714 (+ 55us) server_negotiation.cc:407] Connection header received
0619 12:00:08.490782 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.490783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.490793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.490839 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.491438 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.492007 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.492876 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.493084 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.493149 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.495407 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:08.495409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:08.495409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:08.495416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:08.496988 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.497407 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.497408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.497409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.497467 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.497648 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.497649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.497649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.497739 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:08.497821 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:08.497930 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 12:00:08.497992 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 12:00:08.500061 6872 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:08.499311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.0.203.193:36853)
0619 12:00:08.499352 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.499354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.499484 (+ 130us) server_negotiation.cc:407] Connection header received
0619 12:00:08.499558 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.499559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.499571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.499663 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.500002 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46980: BlockingRecv error: recv got EOF from 127.0.0.1:46980 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 12:00:08.500303 815 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781870380032316-815-0/krb5kdc/krb5cc' of type FILE
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.501574 815 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:08.502571 815 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 12:00:08.511818 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:08.503206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.0.203.193:36853)
0619 12:00:08.503262 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:08.503264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:08.503370 (+ 106us) server_negotiation.cc:407] Connection header received
0619 12:00:08.503458 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:08.503459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:08.503468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:08.503528 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:08.503931 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.504429 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.505332 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:08.505529 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:08.505600 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:08.508220 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:08.508222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:08.508222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:08.508231 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:08.509934 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.510706 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.510708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.510708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.510826 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:08.511070 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:08.511071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:08.511071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:08.511317 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:08.511435 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:08.511602 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:00:08.511687 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:00:08.512723 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6468
I20260619 12:00:08.520578 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6602
I20260619 12:00:08.526024 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6736
I20260619 12:00:08.530877 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6374
2026-06-19T12:00:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2037 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6885](info): set up 2 sockets
Jun 19 12:00:08 dist-test-slave-fbq8 krb5kdc[6885](info): commencing operation
krb5kdc: starting...
W20260619 12:00:09.592939 815 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:09 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab.
Entry for principal oryx/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab.
I20260619 12:00:09.726003 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36661
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab
--principal=oryx/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:36661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:09.807735 6898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:09.807968 6898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:09.807991 6898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:09.809196 6898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:09.809247 6898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:09.809262 6898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:09.809274 6898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:09.809286 6898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:09.810815 6898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:36661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab
--principal=oryx/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:09.811084 6898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:09.811340 6898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:09.816834 6906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:09.816960 6904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:09.816956 6903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:09.817272 6898 server_base.cc:1034] running on GCE node
I20260619 12:00:09.817540 6898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:09.817873 6898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:09.819064 6898 hybrid_clock.cc:648] HybridClock initialized: now 1781870409819028 us; error 66 us; skew 500 ppm
Jun 19 12:00:09 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:09.822104 6898 init.cc:377] Logged in from keytab as oryx/127.0.203.254@KRBTEST.COM (short username oryx)
I20260619 12:00:09.822645 6898 webserver.cc:460] Webserver started at http://127.0.203.254:42529/ using document root <none> and password file <none>
I20260619 12:00:09.822940 6898 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:09.823011 6898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:09.823176 6898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:09.824285 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "4a432676ad224681b060d6872c00e574"
format_stamp: "Formatted at 2026-06-19 12:00:09 on dist-test-slave-fbq8"
server_key: "d935add9ddcd1d83f15ec8364a0bb615"
server_key_iv: "f3d920f5b0959d802840d7473d74e777"
server_key_version: "encryptionkey@0"
I20260619 12:00:09.824707 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "4a432676ad224681b060d6872c00e574"
format_stamp: "Formatted at 2026-06-19 12:00:09 on dist-test-slave-fbq8"
server_key: "d935add9ddcd1d83f15ec8364a0bb615"
server_key_iv: "f3d920f5b0959d802840d7473d74e777"
server_key_version: "encryptionkey@0"
I20260619 12:00:09.826182 6898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:09.827134 6913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:09.827553 6898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:09.827708 6898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "4a432676ad224681b060d6872c00e574"
format_stamp: "Formatted at 2026-06-19 12:00:09 on dist-test-slave-fbq8"
server_key: "d935add9ddcd1d83f15ec8364a0bb615"
server_key_iv: "f3d920f5b0959d802840d7473d74e777"
server_key_version: "encryptionkey@0"
I20260619 12:00:09.827822 6898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:09.861142 6898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:09.863263 6898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:09.863441 6898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:09.867973 6898 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:36661
I20260619 12:00:09.868047 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:36661 every 8 connection(s)
I20260619 12:00:09.868446 6898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:09.869621 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:09.872157 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6898
I20260619 12:00:09.872304 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:09.872465 6966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: Bootstrap starting.
I20260619 12:00:09.872541 815 external_mini_cluster.cc:1428] Setting key f31f87f3f7e737a9db74e21c60219c3f
I20260619 12:00:09.873324 6966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:09.873719 6966 log.cc:824] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:09.874559 6966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: No bootstrap required, opened a new log
I20260619 12:00:09.877063 6966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } }
I20260619 12:00:09.877250 6966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:09.877285 6966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a432676ad224681b060d6872c00e574, State: Initialized, Role: FOLLOWER
I20260619 12:00:09.877456 6966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } }
I20260619 12:00:09.877571 6966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:09.877620 6966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:09.877673 6966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:09.878405 6966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } }
I20260619 12:00:09.878552 6966 leader_election.cc:302] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a432676ad224681b060d6872c00e574; no voters:
I20260619 12:00:09.878762 6966 leader_election.cc:288] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:09.878875 6971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:09.879048 6966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:09.879236 6971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 1 LEADER]: Becoming Leader. State: Replica: 4a432676ad224681b060d6872c00e574, State: Running, Role: LEADER
I20260619 12:00:09.879458 6971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } }
I20260619 12:00:09.880192 6972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } }
I20260619 12:00:09.880297 6972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:09.880540 6973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a432676ad224681b060d6872c00e574. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } }
I20260619 12:00:09.880609 6973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
Jun 19 12:00:09 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
W20260619 12:00:09.881381 6987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:00:09.881467 6987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:00:09.881520 6982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:09.881811 6982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:09.883843 6982 catalog_manager.cc:1347] Generated new cluster ID: c75bda7ddbe2420f95092867a6a000a5
I20260619 12:00:09.883903 6982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:09.885739 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:09.873242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.0.203.254:36661)
0619 12:00:09.873438 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:09.873457 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:09.873487 (+ 30us) server_negotiation.cc:407] Connection header received
0619 12:00:09.874157 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:09.874165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:09.874217 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:09.874356 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:09.875229 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:09.876186 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:09.877423 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:09.877988 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:09.878089 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:09.881334 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:09.881339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:09.881339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:09.881355 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:09.883755 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:09.884236 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:09.884241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:09.884241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:09.884395 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:09.884685 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:09.884686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:09.884686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:09.885031 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:09.885191 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:09.885311 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 12:00:09.885495 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260619 12:00:09.900414 6982 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:09.901087 6982 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:09.907631 6982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: Generated new TSK 0
I20260619 12:00:09.907835 6982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab.
Entry for principal oryx/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab.
I20260619 12:00:09.943637 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36661
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab
--principal=oryx/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:10.041221 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:10.041491 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:10.041518 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:10.042680 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:10.042717 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:10.042744 6992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:10.044169 6992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab
--principal=oryx/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:10.044384 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:10.044586 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:10.050307 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.050319 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.050307 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:10.050726 6992 server_base.cc:1034] running on GCE node
I20260619 12:00:10.051028 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:10.051317 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:10.052505 6992 hybrid_clock.cc:648] HybridClock initialized: now 1781870410052483 us; error 35 us; skew 500 ppm
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:10.055668 6992 init.cc:377] Logged in from keytab as oryx/127.0.203.193@KRBTEST.COM (short username oryx)
I20260619 12:00:10.056260 6992 webserver.cc:460] Webserver started at http://127.0.203.193:38813/ using document root <none> and password file <none>
I20260619 12:00:10.056530 6992 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:10.056608 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:10.056826 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:10.058048 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "fac5ee34febd4082a0253d11b8b47de0"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "6426a75584ec856da11953659c3068da"
server_key_iv: "60912f264aed820d340a81ee39d46ee3"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.058450 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "fac5ee34febd4082a0253d11b8b47de0"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "6426a75584ec856da11953659c3068da"
server_key_iv: "60912f264aed820d340a81ee39d46ee3"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.060045 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 12:00:10.061060 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.061427 6992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.061569 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "fac5ee34febd4082a0253d11b8b47de0"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "6426a75584ec856da11953659c3068da"
server_key_iv: "60912f264aed820d340a81ee39d46ee3"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.061656 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:10.084213 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:10.087188 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:10.087360 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:10.087652 6992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:10.088027 6992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:10.088083 6992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.088121 6992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:10.088150 6992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.095808 6992 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:44299
I20260619 12:00:10.095856 7120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:44299 every 8 connection(s)
I20260619 12:00:10.096226 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:10.099956 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 6992
I20260619 12:00:10.100104 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:10.100334 815 external_mini_cluster.cc:1428] Setting key 4e0c8d7faec6af478b33794fb61a42f0
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:10.108116 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.097084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:44701 (local address 127.0.203.254:36661)
0619 12:00:10.097128 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.097135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.097396 (+ 261us) server_negotiation.cc:407] Connection header received
0619 12:00:10.097981 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.097982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.097994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.098080 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.099150 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.099770 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.101212 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.101449 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.101564 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.104513 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.104515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.104516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.104529 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.106359 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.106947 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.106948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.106948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.107015 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.107365 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.107366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.107366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.107492 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.107555 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.107955 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.108042 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 12:00:10.108389 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.097029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:44701)
0619 12:00:10.097216 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.097262 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.097877 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.098173 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.098177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.098781 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.098998 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.099002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.099903 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.099904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.101119 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.101122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.101569 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.101570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.101726 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.102386 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.102396 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.104382 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.106469 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.106476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.106479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.106833 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.107113 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.107114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.107114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.107280 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.107595 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.107602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.107789 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.107932 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.108114 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260619 12:00:10.109037 7121 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36661
I20260619 12:00:10.109196 7121 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:10.109468 7121 heartbeater.cc:511] Master 127.0.203.254:36661 requested a full tablet report, sending...
I20260619 12:00:10.110118 6930 ts_manager.cc:194] Registered new tserver with Master: fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193:44299)
WARNING: no policy specified for oryx/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 12:00:10.110783 6930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:44701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab.
Entry for principal oryx/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab.
I20260619 12:00:10.132321 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36661
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab
--principal=oryx/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:10.220269 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:10.220557 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:10.220602 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:10.221884 7126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:10.221967 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:10.222015 7126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:10.223510 7126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab
--principal=oryx/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:10.223803 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:10.224048 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:10.229717 7132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.229749 7134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.229717 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:10.230196 7126 server_base.cc:1034] running on GCE node
I20260619 12:00:10.230460 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:10.230777 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:10.231956 7126 hybrid_clock.cc:648] HybridClock initialized: now 1781870410231941 us; error 43 us; skew 500 ppm
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:10.234944 7126 init.cc:377] Logged in from keytab as oryx/127.0.203.194@KRBTEST.COM (short username oryx)
I20260619 12:00:10.235646 7126 webserver.cc:460] Webserver started at http://127.0.203.194:41577/ using document root <none> and password file <none>
I20260619 12:00:10.236107 7126 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:10.236179 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:10.236338 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:10.237715 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "df0ef59ef3684dc5b7737aaeab902ded"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "3f4081ace1ba237aa5093c9af6a1c9e6"
server_key_iv: "9b95f9efbd376148402ebff058c9be25"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.238157 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "df0ef59ef3684dc5b7737aaeab902ded"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "3f4081ace1ba237aa5093c9af6a1c9e6"
server_key_iv: "9b95f9efbd376148402ebff058c9be25"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.239751 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 12:00:10.240824 7141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.241101 7126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.241216 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "df0ef59ef3684dc5b7737aaeab902ded"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "3f4081ace1ba237aa5093c9af6a1c9e6"
server_key_iv: "9b95f9efbd376148402ebff058c9be25"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.241297 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:10.251528 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:10.253943 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:10.254225 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:10.254537 7126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:10.254954 7126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:10.255018 7126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.255056 7126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:10.255079 7126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.262655 7126 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:35073
I20260619 12:00:10.262701 7254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:35073 every 8 connection(s)
I20260619 12:00:10.263059 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:10.268602 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 7126
I20260619 12:00:10.268790 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:10.269030 815 external_mini_cluster.cc:1428] Setting key 156aab86cb9009508f2316b0dc8be3cc
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:10.274362 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.263914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:38843 (local address 127.0.203.254:36661)
0619 12:00:10.263988 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.263990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.264224 (+ 234us) server_negotiation.cc:407] Connection header received
0619 12:00:10.264920 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.264922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.264934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.265020 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.266141 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.266676 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.267771 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.267960 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.268035 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.270999 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.271001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.271002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.271011 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.272854 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.273346 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.273347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.273348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.273410 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.273675 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.273676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.273676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.273800 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.273886 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.274169 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.274260 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:00:10.274590 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.263863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:38843)
0619 12:00:10.264038 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.264084 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.264811 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.265102 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.265106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.265756 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.266032 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.266037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.266786 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.266787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.267684 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.267686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.268078 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.268079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.268208 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.268974 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.268984 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.270877 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.272974 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.272984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.272987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.273262 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.273488 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.273489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.273489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.273602 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.273868 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.273874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.274036 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.274151 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.274349 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260619 12:00:10.275377 7255 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36661
I20260619 12:00:10.275475 7255 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:10.275743 7255 heartbeater.cc:511] Master 127.0.203.254:36661 requested a full tablet report, sending...
I20260619 12:00:10.276309 6930 ts_manager.cc:194] Registered new tserver with Master: df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194:35073)
I20260619 12:00:10.276831 6930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:38843
WARNING: no policy specified for oryx/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab.
Entry for principal oryx/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab.
I20260619 12:00:10.303539 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36661
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab
--principal=oryx/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:10.399115 7260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:10.399380 7260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:10.399406 7260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:10.400785 7260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:10.400842 7260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:10.400872 7260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:10.402645 7260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab
--principal=oryx/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:36661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:10.402941 7260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:10.403311 7260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:10.410269 7268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.410269 7266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.410578 7265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:10.411511 7260 server_base.cc:1034] running on GCE node
I20260619 12:00:10.411952 7260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:10.412456 7260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:10.414144 7260 hybrid_clock.cc:648] HybridClock initialized: now 1781870410414056 us; error 96 us; skew 500 ppm
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:10.418196 7260 init.cc:377] Logged in from keytab as oryx/127.0.203.195@KRBTEST.COM (short username oryx)
I20260619 12:00:10.418758 7260 webserver.cc:460] Webserver started at http://127.0.203.195:44321/ using document root <none> and password file <none>
I20260619 12:00:10.419034 7260 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:10.419111 7260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:10.419438 7260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:10.420691 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "932fa8f21d0f48a1b19749a2c7858864"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "5028f9e15cf30acbe061423b5e8c120e"
server_key_iv: "e212cc3cc25d4cbc326ab8086bde13fe"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.421146 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "932fa8f21d0f48a1b19749a2c7858864"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "5028f9e15cf30acbe061423b5e8c120e"
server_key_iv: "e212cc3cc25d4cbc326ab8086bde13fe"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.422752 7260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:10.423882 7275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.424175 7260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.424321 7260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "932fa8f21d0f48a1b19749a2c7858864"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "5028f9e15cf30acbe061423b5e8c120e"
server_key_iv: "e212cc3cc25d4cbc326ab8086bde13fe"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.424434 7260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:10.439311 7260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:10.441924 7260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:10.442118 7260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:10.442405 7260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:10.442775 7260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:10.442844 7260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.442890 7260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:10.442942 7260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.450254 7260 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:44983
I20260619 12:00:10.450301 7388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:44983 every 8 connection(s)
I20260619 12:00:10.450670 7260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:10.460117 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 7260
I20260619 12:00:10.460242 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:10.460461 815 external_mini_cluster.cc:1428] Setting key 7a02d3cb76d920e1ca4b681174a63824
I20260619 12:00:10.462196 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.451513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:49827 (local address 127.0.203.254:36661)
0619 12:00:10.451638 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.451642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.451872 (+ 230us) server_negotiation.cc:407] Connection header received
0619 12:00:10.452557 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.452558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.452572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.452627 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.453782 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.454338 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.455396 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.455621 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.455710 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.458535 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.458536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.458536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.458545 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.460219 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.461029 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.461031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.461031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.461133 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.461454 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.461455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.461456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.461618 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.461702 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.462007 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.462090 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 12:00:10.462464 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.451452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:49827)
0619 12:00:10.451673 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.451736 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.452411 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.452737 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.452741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.453430 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.453665 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.453669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.454435 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.454436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.455298 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.455300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.455719 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.455720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.455855 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.456477 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.456487 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.458402 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.460385 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.460394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.460397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.460884 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.461216 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.461217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.461217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.461354 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.461692 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.461698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.461850 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.461975 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.462166 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260619 12:00:10.463065 7389 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36661
I20260619 12:00:10.463191 7389 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:10.463358 7389 heartbeater.cc:511] Master 127.0.203.254:36661 requested a full tablet report, sending...
I20260619 12:00:10.463754 6930 ts_manager.cc:194] Registered new tserver with Master: 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983)
I20260619 12:00:10.464141 6930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:49827
I20260619 12:00:10.472680 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.203.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.253.keytab.
Entry for principal oryx/127.0.203.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.253.keytab.
I20260619 12:00:10.503531 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:46427
--webserver_interface=127.0.203.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.253.keytab
--principal=oryx/127.0.203.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:36661,127.0.203.253:46427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:10.592319 7394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:10.592543 7394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:10.592566 7394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:10.593834 7394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:10.593889 7394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:10.593904 7394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:10.593917 7394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:10.593930 7394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:10.595335 7394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:36661,127.0.203.253:46427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.253.keytab
--principal=oryx/127.0.203.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:46427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
W20260619 12:00:10.595440 7394 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.203.254:36661,127.0.203.253:46427'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 12:00:10.595556 7394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:10.595737 7394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:10.601061 7400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.601080 7402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.601063 7399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:10.601814 7394 server_base.cc:1034] running on GCE node
I20260619 12:00:10.602232 7394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:10.602632 7394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:10.603827 7394 hybrid_clock.cc:648] HybridClock initialized: now 1781870410603793 us; error 46 us; skew 500 ppm
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:10.606948 7394 init.cc:377] Logged in from keytab as oryx/127.0.203.253@KRBTEST.COM (short username oryx)
I20260619 12:00:10.607535 7394 webserver.cc:460] Webserver started at http://127.0.203.253:34719/ using document root <none> and password file <none>
I20260619 12:00:10.607825 7394 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:10.607894 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:10.608053 7394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:10.609391 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/instance:
uuid: "0c06179b2a2c4090b9bed40befcfaebd"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "4fef8478f65a4f1adac2e3b671c9982b"
server_key_iv: "18e0680d66e6bf7a1c9fbfbc111fe616"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.609798 7394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal/instance:
uuid: "0c06179b2a2c4090b9bed40befcfaebd"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "4fef8478f65a4f1adac2e3b671c9982b"
server_key_iv: "18e0680d66e6bf7a1c9fbfbc111fe616"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.611406 7394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:10.612387 7409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.612727 7394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.612843 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
uuid: "0c06179b2a2c4090b9bed40befcfaebd"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "4fef8478f65a4f1adac2e3b671c9982b"
server_key_iv: "18e0680d66e6bf7a1c9fbfbc111fe616"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.612937 7394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:10.631727 7394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:10.634276 7394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:10.634454 7394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:10.639137 7394 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.253:46427
I20260619 12:00:10.639204 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.253:46427 every 8 connection(s)
I20260619 12:00:10.639626 7394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
I20260619 12:00:10.640808 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:10.642388 7462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.253@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:10.649560 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 7394
I20260619 12:00:10.652745 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.643182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.203.254:36661)
0619 12:00:10.643266 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.643270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.643505 (+ 235us) server_negotiation.cc:407] Connection header received
0619 12:00:10.644041 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.644042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.644053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.644125 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.644865 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.645458 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.646594 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.646807 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.646908 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.649262 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.649264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.649264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.649276 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.650976 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.651527 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.651528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.651528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.651587 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.651925 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.651926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.651926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.652104 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.652190 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.652520 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.652614 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:00:10.652971 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.643118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:33992)
0619 12:00:10.643325 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.643375 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.643874 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.644198 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.644204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.644500 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.644772 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.644777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.645544 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.645545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.646400 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.646403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.646942 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.646943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.647070 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.647647 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.647656 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.649145 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.651084 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.651090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.651093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.651418 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.651694 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.651696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.651696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.651850 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.652190 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.652207 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.652370 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.652486 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.652711 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260619 12:00:10.653864 7462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.253@KRBTEST.COM
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.253@KRBTEST.COM for oryx/127.0.203.253@KRBTEST.COM
I20260619 12:00:10.659619 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.650125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.0.203.253:46427)
0619 12:00:10.650616 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.650621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.650638 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:00:10.650657 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.650659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.650670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.650764 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.651532 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.652222 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.653222 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.653664 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.653725 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.656161 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.656170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.656170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.656179 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.658246 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.658674 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.658678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.658678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.658750 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.658966 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.658967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.658967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.659137 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.659340 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.659384 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.659470 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":411,"threads_started":1}
I20260619 12:00:10.661914 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.654207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.0.1:39558)
0619 12:00:10.654255 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.654265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.654300 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.654616 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.654617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.654777 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.654824 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.654825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.655505 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.655505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.656051 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.656053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.656434 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.656434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.656514 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.656967 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.656971 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.658501 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.660957 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.660958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.660958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.661114 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.661309 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.661309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.661309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.661386 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.661631 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.661632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.661672 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.661756 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.661844 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 12:00:10.662072 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.654278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.0.203.253:46427)
0619 12:00:10.654464 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.654465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.654471 (+ 6us) server_negotiation.cc:407] Connection header received
0619 12:00:10.654485 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.654486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.654494 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.654594 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.654912 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.655418 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.656125 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.656331 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.656436 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.658601 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.658602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.658602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.658609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.660874 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.661193 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.661194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.661194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.661244 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.661443 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.661443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.661444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.661537 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.661594 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.661771 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.661894 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":166,"thread_start_us":118,"threads_started":1}
I20260619 12:00:10.663623 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Bootstrap starting.
I20260619 12:00:10.664417 7462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:10.664809 7462 log.cc:824] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:10.665719 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: No bootstrap required, opened a new log
I20260619 12:00:10.667311 7462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } }
I20260619 12:00:10.667465 7462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:10.667491 7462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c06179b2a2c4090b9bed40befcfaebd, State: Initialized, Role: FOLLOWER
I20260619 12:00:10.667634 7462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } }
I20260619 12:00:10.668007 7462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:10.667972 7469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } } }
I20260619 12:00:10.668164 7469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: FOLLOWER
W20260619 12:00:10.670190 7480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:00:10.670264 7480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:00:10.672458 7394 master_runner.cc:428] Detected that this master 0c06179b2a2c4090b9bed40befcfaebd is joining an existing cluster
I20260619 12:00:10.672514 7394 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.203.253:46427
I20260619 12:00:10.680548 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.673832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.0.203.254:36661)
0619 12:00:10.673877 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.673879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.673887 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:10.673935 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.673936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.673946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.673995 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.674395 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.674916 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.675754 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.675959 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.676045 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.677171 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.677172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.677173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.677180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.679031 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.679531 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.679532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.679533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.679597 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.679884 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.679885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.679885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.680038 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.680123 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.680344 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.680422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:00:10.681006 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.673633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34002)
0619 12:00:10.673802 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.673815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.673873 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.674001 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.674002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.674249 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.674324 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.674326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.675001 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.675002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.675638 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.675640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.676059 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.676059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.676133 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.676683 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.676689 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.677060 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.679155 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.679156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.679156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.679439 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.679731 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.679731 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.679732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.679802 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.680129 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.680131 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.680216 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.680322 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.680465 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260619 12:00:10.681372 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.673681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.0.203.253:46427)
0619 12:00:10.673727 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.673729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.674001 (+ 272us) server_negotiation.cc:407] Connection header received
0619 12:00:10.674107 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.674108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.674118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.674182 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.674697 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.675478 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.676519 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.676849 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.676893 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.678246 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.678248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.678249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.678257 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.680102 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.680490 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.680491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.680491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.680538 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.680809 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.680810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.680810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.680923 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.681003 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.681162 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.681255 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:00:10.681808 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.673785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.0.1:39564)
0619 12:00:10.673935 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.673945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.674024 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.674303 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.674304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.674516 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.674594 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.674596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.675575 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.675576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.676426 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.676428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.676968 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.676969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.677026 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.677723 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.677729 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.678152 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.680186 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.680187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.680187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.680415 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.680600 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.680601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.680601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.680733 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.680995 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.680995 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.681036 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.681142 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.681717 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":63,"thread_start_us":78,"threads_started":1}
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.254@KRBTEST.COM for oryx/127.0.203.253@KRBTEST.COM
I20260619 12:00:10.692195 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.683691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.203.253:46427)
0619 12:00:10.683737 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.683739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.683932 (+ 193us) server_negotiation.cc:407] Connection header received
0619 12:00:10.684015 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.684015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.684023 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.684074 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.684708 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.685234 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.686401 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.686589 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.686654 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.689154 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.689155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.689155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.689163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.690631 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.691261 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.691262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.691263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.691332 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.691553 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.691554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.691554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.691679 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.691755 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.691994 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.692089 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:00:10.692209 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.683599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.0.1:39574)
0619 12:00:10.683786 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.683847 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.683937 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.684088 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.684092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.684456 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.684536 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.684539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.685330 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.685331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.686309 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.686312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.686675 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.686676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.686764 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.687240 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.687245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.689036 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.690737 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.690748 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.690750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.691062 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.691411 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.691412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.691413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.691483 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.691758 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.691767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.691833 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.691928 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.692101 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260619 12:00:10.692982 6930 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.203.253:46427: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.203.253"
port: 46427
}
attrs {
promote: true
}
}
dest_uuid: "4a432676ad224681b060d6872c00e574"
cas_config_opid_index: -1
I20260619 12:00:10.693339 6930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } }
I20260619 12:00:10.693908 7489 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0c06179b2a2c4090b9bed40befcfaebd (127.0.203.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } }
I20260619 12:00:10.694239 7436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:10.694945 7489 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.203.253:46427
I20260619 12:00:10.695041 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } }
I20260619 12:00:10.695106 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:10.695359 7436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Refusing update from remote peer 4a432676ad224681b060d6872c00e574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 12:00:10.695785 7394 master.cc:537] Master@127.0.203.253:46427 shutting down...
I20260619 12:00:10.695854 7489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 12:00:10.697118 6916 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 12:00:10.697767 6916 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 -> Peer 0c06179b2a2c4090b9bed40befcfaebd (127.0.203.253:46427): Couldn't send request to peer 0c06179b2a2c4090b9bed40befcfaebd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:00:10.699427 7394 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:10.699561 7394 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:10.699616 7394 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: stopping tablet replica
I20260619 12:00:10.713194 7394 master.cc:552] Master@127.0.203.253:46427 shutdown complete.
I20260619 12:00:10.713277 7394 master_runner.cc:305] Clearing existing system tablet
I20260619 12:00:10.713418 7394 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:10.713482 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:10.714226 7394 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:10.714771 7494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.715050 7394 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:10.715134 7394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.715164 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
uuid: "0c06179b2a2c4090b9bed40befcfaebd"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "4fef8478f65a4f1adac2e3b671c9982b"
server_key_iv: "18e0680d66e6bf7a1c9fbfbc111fe616"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.715860 7394 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:10.717398 7394 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 12:00:10.717509 7394 log.cc:1196] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 12:00:10.717770 7394 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Deleting consensus metadata
I20260619 12:00:10.717911 7394 master_runner.cc:315] Copying system tablet from 127.0.203.254:36661
I20260619 12:00:10.718588 7394 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.254:36661
I20260619 12:00:10.726629 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.718995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34008)
0619 12:00:10.719240 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:10.719251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:10.719338 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:10.719508 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:10.719509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:10.719695 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:10.719758 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.719760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.720899 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.720900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.721498 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:10.721499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.722036 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.722037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:10.722118 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.722796 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:10.722801 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:10.723126 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:10.725354 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.725355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.725355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.725589 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.725847 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:10.725848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:10.725849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:10.725915 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:10.726304 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:10.726305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:10.726357 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 12:00:10.726433 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 12:00:10.726535 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260619 12:00:10.726672 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.719075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.0.203.254:36661)
0619 12:00:10.719137 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.719163 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.719356 (+ 193us) server_negotiation.cc:407] Connection header received
0619 12:00:10.719402 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.719403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.719419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.719495 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.719842 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.720776 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.721600 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.721925 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.722005 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.723230 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.723231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.723232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.723240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.725225 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.725680 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.725681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.725681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.725751 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.725999 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.725999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.726000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.726145 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.726221 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.726490 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.726573 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 12:00:10.727474 6950 tablet_copy_service.cc:140] P 4a432676ad224681b060d6872c00e574: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0c06179b2a2c4090b9bed40befcfaebd ({username='oryx', principal='oryx/127.0.203.253@KRBTEST.COM'} at 127.0.0.1:34008)
I20260619 12:00:10.727602 6950 tablet_copy_service.cc:161] P 4a432676ad224681b060d6872c00e574: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0c06179b2a2c4090b9bed40befcfaebd at {username='oryx', principal='oryx/127.0.203.253@KRBTEST.COM'} at 127.0.0.1:34008: session id = 0c06179b2a2c4090b9bed40befcfaebd-00000000000000000000000000000000
I20260619 12:00:10.728305 6950 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:00:10.729003 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 12:00:10.730002 7394 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: tablet copy: Starting download of 0 data blocks...
I20260619 12:00:10.730175 7394 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: tablet copy: Starting download of 1 WAL segments...
I20260619 12:00:10.731333 7394 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:00:10.732256 6950 tablet_copy_service.cc:342] P 4a432676ad224681b060d6872c00e574: Request end of tablet copy session 0c06179b2a2c4090b9bed40befcfaebd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.203.253@KRBTEST.COM'} at 127.0.0.1:34008
I20260619 12:00:10.732321 6950 tablet_copy_service.cc:433] P 4a432676ad224681b060d6872c00e574: ending tablet copy session 0c06179b2a2c4090b9bed40befcfaebd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0c06179b2a2c4090b9bed40befcfaebd
I20260619 12:00:10.735805 7394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:10.737322 7502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.737360 7501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:10.737983 7394 server_base.cc:1034] running on GCE node
W20260619 12:00:10.740960 7504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:10.741189 7394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:10.741400 7394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:10.742606 7394 hybrid_clock.cc:648] HybridClock initialized: now 1781870410742586 us; error 46 us; skew 500 ppm
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:10.744423 7394 init.cc:377] Logged in from keytab as oryx/127.0.203.253@KRBTEST.COM (short username oryx)
I20260619 12:00:10.744870 7394 webserver.cc:460] Webserver started at http://127.0.203.253:43623/ using document root <none> and password file <none>
I20260619 12:00:10.745008 7394 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:10.745049 7394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:10.745929 7394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.746585 7511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.746788 7394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.746855 7394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
uuid: "0c06179b2a2c4090b9bed40befcfaebd"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "4fef8478f65a4f1adac2e3b671c9982b"
server_key_iv: "18e0680d66e6bf7a1c9fbfbc111fe616"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.746927 7394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:10.788164 7394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:10.790037 7394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:10.794029 7394 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.253:46427
I20260619 12:00:10.794431 7563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.253:46427 every 8 connection(s)
I20260619 12:00:10.794487 7394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
I20260619 12:00:10.795399 7564 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:00:10.795902 7564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Bootstrap starting.
I20260619 12:00:10.798599 7564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:00:10.798822 7564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Bootstrap complete.
I20260619 12:00:10.799031 7564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } }
I20260619 12:00:10.799120 7564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0c06179b2a2c4090b9bed40befcfaebd, State: Initialized, Role: LEARNER
I20260619 12:00:10.799178 7564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } }
I20260619 12:00:10.799394 7564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:10.799392 7567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } }
I20260619 12:00:10.799557 7567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: LEARNER
I20260619 12:00:10.801831 7578 catalog_manager.cc:1259] Loaded cluster ID: c75bda7ddbe2420f95092867a6a000a5
I20260619 12:00:10.801911 7578 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: loading cluster ID for follower catalog manager: success
I20260619 12:00:10.803073 7578 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: acquiring CA information for follower catalog manager: success
I20260619 12:00:10.803443 7578 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:00:10.820591 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:10.812265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.0.203.253:46427)
0619 12:00:10.812450 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:10.812453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:10.812467 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:00:10.812503 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:10.812504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:10.812519 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:10.812619 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:10.813387 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.814479 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.815352 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:10.815690 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:10.815775 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:10.817308 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:10.817309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:10.817310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:10.817317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:10.819070 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.819500 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.819502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.819502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.819571 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:10.819818 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:10.819819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:10.819820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:10.820087 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:10.820225 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:10.820360 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 12:00:10.820455 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.0.203.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.252.keytab.
Entry for principal oryx/127.0.203.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.252.keytab.
I20260619 12:00:10.852334 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:32795
--webserver_interface=127.0.203.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.252.keytab
--principal=oryx/127.0.203.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:10.943938 7582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:10.944166 7582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:10.944188 7582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:10.945462 7582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:10.945513 7582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:10.945526 7582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:10.945539 7582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:10.945551 7582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:10.947082 7582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.252.keytab
--principal=oryx/127.0.203.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:32795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:10.947356 7582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:10.947636 7582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:10.953254 7588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.953249 7590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:10.953244 7587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:10.953524 7582 server_base.cc:1034] running on GCE node
I20260619 12:00:10.954028 7582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:10.954402 7582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:10.955606 7582 hybrid_clock.cc:648] HybridClock initialized: now 1781870410955582 us; error 51 us; skew 500 ppm
Jun 19 12:00:10 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:10.958590 7582 init.cc:377] Logged in from keytab as oryx/127.0.203.252@KRBTEST.COM (short username oryx)
I20260619 12:00:10.959182 7582 webserver.cc:460] Webserver started at http://127.0.203.252:38529/ using document root <none> and password file <none>
I20260619 12:00:10.959440 7582 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:10.959510 7582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:10.959676 7582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:10.960934 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/instance:
uuid: "271f9c8643584928b89cff5a7a9447c7"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "8cf51e40042d1dfa18baba8d54900d98"
server_key_iv: "0bc8ef28a75effe138284b5aa07523a8"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.961314 7582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal/instance:
uuid: "271f9c8643584928b89cff5a7a9447c7"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "8cf51e40042d1dfa18baba8d54900d98"
server_key_iv: "0bc8ef28a75effe138284b5aa07523a8"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.962821 7582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:10.963892 7597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:10.964169 7582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:10.964275 7582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
uuid: "271f9c8643584928b89cff5a7a9447c7"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "8cf51e40042d1dfa18baba8d54900d98"
server_key_iv: "0bc8ef28a75effe138284b5aa07523a8"
server_key_version: "encryptionkey@0"
I20260619 12:00:10.964365 7582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:10.989712 7582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:10.992318 7582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:10.992506 7582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:10.998047 7582 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.252:32795
I20260619 12:00:10.998193 7649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.252:32795 every 8 connection(s)
I20260619 12:00:10.998778 7582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
I20260619 12:00:10.999972 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:11.002038 7650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:00:11.008793 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 7582
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.252@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:11.013774 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.003190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.0.203.254:36661)
0619 12:00:11.003290 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.003292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.003307 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:00:11.003977 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.003979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.003992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.004047 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.004888 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.005426 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.007237 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.007439 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.007515 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.010628 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.010629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.010630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.010638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.012226 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.012743 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.012745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.012745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.012810 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.013077 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.013078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.013078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.013206 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.013286 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.013579 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.013668 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:00:11.013991 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.002931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34014)
0619 12:00:11.003142 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.003191 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.003851 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.004146 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.004150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.004458 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.004764 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.004772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.005521 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.005523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.007125 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.007128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.007543 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.007545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.007682 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.008323 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.008333 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.010519 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.012354 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.012361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.012364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.012616 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.012881 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.012882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.012883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.012996 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.013279 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.013285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.013433 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.013548 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.013725 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":117,"threads_started":1}
I20260619 12:00:11.014775 7650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.252@KRBTEST.COM
I20260619 12:00:11.019259 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.009461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.0.203.252:32795)
0619 12:00:11.009921 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.009928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.009957 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:00:11.009987 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.009989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.010009 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.010130 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.010800 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.011854 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.012597 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.013010 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.013097 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.016213 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.016221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.016222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.016228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.017934 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.018376 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.018380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.018380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.018444 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.018654 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.018655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.018656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.018807 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.018970 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.019051 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.019139 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":347,"threads_started":1}
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.252@KRBTEST.COM for oryx/127.0.203.253@KRBTEST.COM
I20260619 12:00:11.023154 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.015202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.0.1:39606)
0619 12:00:11.015263 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.015274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.015318 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.015498 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.015499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.015695 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.015765 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.015767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.016624 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.016625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.017418 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.017420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.017844 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.017845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.017917 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.018313 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.018318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.019723 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.021895 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.021896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.021896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.022076 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.022374 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.022375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.022375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.022439 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.022891 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.022892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.022931 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.022991 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.023086 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 12:00:11.023386 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.015280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.0.203.253:46427)
0619 12:00:11.015342 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.015344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.015356 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:11.015391 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.015392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.015409 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.015481 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.015867 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.016522 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.017519 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.017754 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.017846 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.019858 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.019860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.019860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.019870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.021770 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.022191 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.022193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.022193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.022265 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.022554 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.022555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.022555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.022769 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.022889 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.023074 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.023173 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:00:11.024061 7650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870410, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.252@KRBTEST.COM for oryx/127.0.203.252@KRBTEST.COM
I20260619 12:00:11.032850 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.024500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.0.1:49342)
0619 12:00:11.024565 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.024578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.024626 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.025059 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.025060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.025234 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.025284 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.025286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.026172 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.026173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.026801 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.026804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.027487 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.027488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.027541 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.028061 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.028069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.029644 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.031519 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.031521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.031521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.031709 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.032014 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.032015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.032015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.032081 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.032429 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.032430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.032475 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.032575 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.032777 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 12:00:11.032975 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.024808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.0.203.252:32795)
0619 12:00:11.024875 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.024877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.024890 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:00:11.024908 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.024908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.024917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.024977 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.025391 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.026058 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.027074 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.027332 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.027415 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.029815 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.029817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.029817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.029826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.031421 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.031819 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.031844 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.031845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.031917 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.032162 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.032163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.032190 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.032321 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.032386 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.032616 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.032840 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":38}
I20260619 12:00:11.034627 7650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Bootstrap starting.
I20260619 12:00:11.035717 7650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:11.036027 7650 log.cc:824] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:11.036991 7650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: No bootstrap required, opened a new log
I20260619 12:00:11.038606 7650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } }
I20260619 12:00:11.038766 7650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:11.038837 7650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271f9c8643584928b89cff5a7a9447c7, State: Initialized, Role: FOLLOWER
I20260619 12:00:11.039003 7650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } }
I20260619 12:00:11.039341 7650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:11.039338 7655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } } }
I20260619 12:00:11.039562 7655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 12:00:11.042237 7666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:00:11.042304 7666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:00:11.045182 7582 master_runner.cc:428] Detected that this master 271f9c8643584928b89cff5a7a9447c7 is joining an existing cluster
I20260619 12:00:11.045270 7582 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.203.252:32795
I20260619 12:00:11.054695 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.046765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.0.203.254:36661)
0619 12:00:11.046818 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.046821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.047002 (+ 181us) server_negotiation.cc:407] Connection header received
0619 12:00:11.047107 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.047107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.047119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.047169 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.047681 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.048475 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.049605 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.049900 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.049958 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.051606 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.051607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.051608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.051615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.053263 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.053850 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.053851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.053851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.053913 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.054133 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.054134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.054134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.054272 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.054364 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.054522 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.054593 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:00:11.054709 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.046709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34018)
0619 12:00:11.046924 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.046938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.047024 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.047230 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.047231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.047534 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.047596 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.047598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.048570 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.048571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.049525 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.049527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.050014 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.050014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.050091 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.050646 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.050652 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.051508 (+ 856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.053372 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.053374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.053375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.053780 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.053984 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.053985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.053985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.054063 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.054356 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.054357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.054401 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.054481 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.054595 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260619 12:00:11.055635 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.047372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.0.1:49344)
0619 12:00:11.048044 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.048054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.048132 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.048270 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.048270 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.048464 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.048537 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.048539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.049424 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.049425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.050225 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.050228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.050705 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.050706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.050763 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.051177 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.051182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.051668 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.053930 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.053931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.053932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.054134 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.054373 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.054374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.054374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.054438 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.054710 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.054710 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.055195 (+ 485us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.055308 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.055450 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":622,"threads_started":1}
I20260619 12:00:11.055718 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.047432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.0.203.252:32795)
0619 12:00:11.047472 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.047474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.048111 (+ 637us) server_negotiation.cc:407] Connection header received
0619 12:00:11.048190 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.048190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.048198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.048222 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.048618 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.049336 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.050312 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.050596 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.050635 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.051769 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.051771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.051771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.051779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.053832 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.054214 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.054215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.054216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.054286 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.054503 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.054503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.054504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.054627 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.054668 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.055347 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.055538 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":22}
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870409, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.254@KRBTEST.COM for oryx/127.0.203.252@KRBTEST.COM
I20260619 12:00:11.071512 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6992
I20260619 12:00:11.072350 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.056952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.0.1:49356)
0619 12:00:11.057017 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.057027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.057067 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.057209 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.057210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.057393 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.057450 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.057451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.058124 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.058125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.058988 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.058991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.059358 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.059359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.059433 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.059848 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.059853 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.061373 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.070815 (+ 9442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.070816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.070817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.071087 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.071364 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.071365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.071366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.071439 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.071802 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.071803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.071862 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.071932 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.072178 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 12:00:11.072738 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.057026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.0.203.252:32795)
0619 12:00:11.057061 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.057063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.057072 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:11.057140 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.057140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.057149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.057199 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.057531 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.058037 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.059094 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.059270 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.059334 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.061476 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.061478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.061478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.061486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.070711 (+ 9225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.071199 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.071201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.071201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.071260 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.071556 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.071558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.071558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.071705 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.071770 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.072503 (+ 733us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.072584 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 12:00:11.074945 6930 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.203.252:32795: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "271f9c8643584928b89cff5a7a9447c7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.203.252"
port: 32795
}
attrs {
promote: true
}
}
dest_uuid: "4a432676ad224681b060d6872c00e574"
cas_config_opid_index: 5
I20260619 12:00:11.075313 6930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } }
I20260619 12:00:11.075790 7490 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.076465 7624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:11.077185 7490 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.203.252:32795
I20260619 12:00:11.077257 7490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.077332 7490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.077677 7624 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Refusing update from remote peer 4a432676ad224681b060d6872c00e574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 12:00:11.077994 7489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:11.078514 7582 master.cc:537] Master@127.0.203.252:32795 shutting down...
I20260619 12:00:11.079548 7624 raft_consensus.cc:858] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } }
W20260619 12:00:11.079648 7624 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } }
I20260619 12:00:11.079826 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:44299
--local_ip_for_outbound_sockets=127.0.203.193
--tserver_master_addrs=127.0.203.254:36661
--webserver_port=38813
--webserver_interface=127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab
--principal=oryx/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:00:11.080124 7624 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0c06179b2a2c4090b9bed40befcfaebd (127.0.203.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } }
I20260619 12:00:11.080731 7676 mvcc.cc:205] Tried to move back new op lower bound from 7298541198979534848 to 7298541198866821120. Current Snapshot: MvccSnapshot[applied={T|T < 7298541198879805440}]
I20260619 12:00:11.082732 7655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a432676ad224681b060d6872c00e574. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } }
I20260619 12:00:11.082837 7655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 12:00:11.082878 7655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } }
I20260619 12:00:11.082926 7655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 12:00:11.082957 7655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } }
I20260619 12:00:11.083001 7655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 12:00:11.083202 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.046786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.0.1:39614)
0619 12:00:11.046942 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.046953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.047007 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.047512 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.047514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.047760 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.047850 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.047852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.049043 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.049044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.050379 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.050383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.050906 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.050907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.050977 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.069298 (+ 18321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.069337 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.069928 (+ 591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.081762 (+ 11834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.081763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.081764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.082072 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.082360 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.082361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.082362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.082435 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.082725 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.082726 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.082762 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.082845 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.082994 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260619 12:00:11.083295 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.047002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.0.203.253:46427)
0619 12:00:11.047048 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.047050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.047061 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:11.047087 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.047088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.047100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.047178 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.048056 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.048942 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.050489 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.050778 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.050821 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.070084 (+ 19263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.070086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.070087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.070099 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.071989 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.082195 (+ 10206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.082197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.082198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.082278 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.082517 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.082518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.082519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.082652 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.082688 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.083085 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.083181 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 12:00:11.086539 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.086659 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.087215 7582 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260619 12:00:11.087265 6914 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 -> Peer 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252:32795): Couldn't send request to peer 271f9c8643584928b89cff5a7a9447c7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:00:11.087334 7582 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 12:00:11.087371 7582 tablet_replica.cc:330] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: stopping tablet replica
I20260619 12:00:11.088020 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.075994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.0.203.253:46427)
0619 12:00:11.078317 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.078319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.078326 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:00:11.078353 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.078354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.078379 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.083849 (+ 5470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:11.083885 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.084695 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.086730 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.087443 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.087530 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.087607 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.087888 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.087948 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2300,"thread_start_us":64,"threads_started":1}
I20260619 12:00:11.088095 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.075890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.0.1:39616)
0619 12:00:11.078051 (+ 2161us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.078067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.078211 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.083266 (+ 5055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.083267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.083271 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:11.083330 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.083332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.084805 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.084806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.086052 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.086055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.087679 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.087680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.087762 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.087791 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.087862 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.087962 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137}
I20260619 12:00:11.088514 7538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 LEARNER]: Refusing update from remote peer 4a432676ad224681b060d6872c00e574: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 12:00:11.088891 7490 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 12:00:11.089540 7538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.091073 7567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a432676ad224681b060d6872c00e574. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.091257 7567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: LEARNER
I20260619 12:00:11.091060 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: NON_VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: true } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.091378 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: LEARNER
I20260619 12:00:11.099326 7489 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: attempting to promote NON_VOTER 0c06179b2a2c4090b9bed40befcfaebd to VOTER
I20260619 12:00:11.099630 7489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } }
I20260619 12:00:11.100503 7538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 LEARNER]: Refusing update from remote peer 4a432676ad224681b060d6872c00e574: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 12:00:11.100555 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.100416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.0.1:49362)
0619 12:00:11.100469 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.100518 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 12:00:11.100849 7490 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 12:00:11.101394 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.101199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.0.1:49364)
0619 12:00:11.101310 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.101358 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 12:00:11.101526 6914 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 -> Peer 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252:32795): Couldn't send request to peer 271f9c8643584928b89cff5a7a9447c7. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.252:32795: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 12:00:11.101996 7489 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c06179b2a2c4090b9bed40befcfaebd (127.0.203.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.102752 7582 master.cc:552] Master@127.0.203.252:32795 shutdown complete.
I20260619 12:00:11.102787 7582 master_runner.cc:305] Clearing existing system tablet
I20260619 12:00:11.102892 7582 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:11.102921 7582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:11.103199 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.103303 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.103289 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.103503 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.103731 7582 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:11.103782 7538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c06179b2a2c4090b9bed40befcfaebd (127.0.203.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.104956 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.105090 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:00:11.105525 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.105659 7582 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:11.105726 7582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:00:11.105767 7582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
uuid: "271f9c8643584928b89cff5a7a9447c7"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "8cf51e40042d1dfa18baba8d54900d98"
server_key_iv: "0bc8ef28a75effe138284b5aa07523a8"
server_key_version: "encryptionkey@0"
I20260619 12:00:11.106639 7582 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:11.108011 7582 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 12:00:11.108090 7582 log.cc:1196] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 12:00:11.108322 7582 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Deleting consensus metadata
I20260619 12:00:11.108566 7582 master_runner.cc:315] Copying system tablet from 127.0.203.254:36661
I20260619 12:00:11.109390 7582 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.203.254:36661
I20260619 12:00:11.122138 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.113026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.0.203.254:36661)
0619 12:00:11.113078 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.113081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.113512 (+ 431us) server_negotiation.cc:407] Connection header received
0619 12:00:11.113689 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.113691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.113704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.113738 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.114311 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.115133 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.116902 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.117224 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.117335 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.118627 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.118628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.118629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.118639 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.120453 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.120943 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.120945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.120945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.121019 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.121248 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.121249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.121249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.121504 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.121638 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.121930 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.122031 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:00:11.122198 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.113145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34034)
0619 12:00:11.113414 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.113426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.113546 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.113816 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.113817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.114090 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.114178 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.114181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.115261 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.115262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.116627 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.116630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.117343 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.117344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.117424 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.118047 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.118053 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.118511 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.120558 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.120559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.120560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.120848 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.121103 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.121104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.121104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.121173 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.121674 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.121676 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.121751 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.121895 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.122059 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260619 12:00:11.122965 6950 tablet_copy_service.cc:140] P 4a432676ad224681b060d6872c00e574: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 271f9c8643584928b89cff5a7a9447c7 ({username='oryx', principal='oryx/127.0.203.252@KRBTEST.COM'} at 127.0.0.1:34034)
I20260619 12:00:11.123041 6950 tablet_copy_service.cc:161] P 4a432676ad224681b060d6872c00e574: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 271f9c8643584928b89cff5a7a9447c7 at {username='oryx', principal='oryx/127.0.203.252@KRBTEST.COM'} at 127.0.0.1:34034: session id = 271f9c8643584928b89cff5a7a9447c7-00000000000000000000000000000000
I20260619 12:00:11.123705 6950 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:00:11.124220 7582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 12:00:11.125965 7582 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: tablet copy: Starting download of 0 data blocks...
I20260619 12:00:11.126135 7582 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: tablet copy: Starting download of 1 WAL segments...
I20260619 12:00:11.127331 7582 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:00:11.128605 6950 tablet_copy_service.cc:342] P 4a432676ad224681b060d6872c00e574: Request end of tablet copy session 271f9c8643584928b89cff5a7a9447c7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.203.252@KRBTEST.COM'} at 127.0.0.1:34034
I20260619 12:00:11.128722 6950 tablet_copy_service.cc:433] P 4a432676ad224681b060d6872c00e574: ending tablet copy session 271f9c8643584928b89cff5a7a9447c7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 271f9c8643584928b89cff5a7a9447c7
I20260619 12:00:11.132206 7582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:11.135556 7698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:11.135670 7582 server_base.cc:1034] running on GCE node
W20260619 12:00:11.137164 7695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:11.140915 7696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:11.141160 7582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:11.141345 7582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:11.142488 7582 hybrid_clock.cc:648] HybridClock initialized: now 1781870411142458 us; error 51 us; skew 500 ppm
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:11.144346 7582 init.cc:377] Logged in from keytab as oryx/127.0.203.252@KRBTEST.COM (short username oryx)
I20260619 12:00:11.144747 7582 webserver.cc:460] Webserver started at http://127.0.203.252:42583/ using document root <none> and password file <none>
I20260619 12:00:11.144902 7582 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:11.144943 7582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:11.145833 7582 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:11.146551 7705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.146762 7582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:11.146847 7582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
uuid: "271f9c8643584928b89cff5a7a9447c7"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "8cf51e40042d1dfa18baba8d54900d98"
server_key_iv: "0bc8ef28a75effe138284b5aa07523a8"
server_key_version: "encryptionkey@0"
I20260619 12:00:11.146974 7582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:11.165644 7582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:11.168223 7582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:11.172614 7582 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.252:32795
I20260619 12:00:11.172904 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.252:32795 every 8 connection(s)
I20260619 12:00:11.173257 7582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
I20260619 12:00:11.174544 7758 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:00:11.175220 7758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Bootstrap starting.
I20260619 12:00:11.178332 7758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:00:11.178593 7758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: Bootstrap complete.
I20260619 12:00:11.178860 7758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } }
I20260619 12:00:11.178960 7758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 271f9c8643584928b89cff5a7a9447c7, State: Initialized, Role: LEARNER
I20260619 12:00:11.178995 7758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } }
I20260619 12:00:11.179235 7758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:11.179239 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.179543 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: LEARNER
I20260619 12:00:11.181082 7772 catalog_manager.cc:1259] Loaded cluster ID: c75bda7ddbe2420f95092867a6a000a5
I20260619 12:00:11.181169 7772 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: loading cluster ID for follower catalog manager: success
I20260619 12:00:11.182641 7772 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: acquiring CA information for follower catalog manager: success
I20260619 12:00:11.183043 7772 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 12:00:11.197929 7682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:11.198144 7682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:11.198170 7682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:11.199805 7682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:11.199906 7682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:11.199963 7682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:11.201890 7682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab
--principal=oryx/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=38813
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:11.202185 7682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:11.202447 7682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:11.209826 7778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:11.209878 7780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:11.209875 7777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:11.210521 7682 server_base.cc:1034] running on GCE node
I20260619 12:00:11.210846 7682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:11.211129 7682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:11.212291 7682 hybrid_clock.cc:648] HybridClock initialized: now 1781870411212266 us; error 46 us; skew 500 ppm
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:11.215191 7682 init.cc:377] Logged in from keytab as oryx/127.0.203.193@KRBTEST.COM (short username oryx)
I20260619 12:00:11.215739 7682 webserver.cc:460] Webserver started at http://127.0.203.193:38813/ using document root <none> and password file <none>
I20260619 12:00:11.215998 7682 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:11.216080 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:11.217598 7682 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:11.218247 7787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.218405 7682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:11.218492 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "fac5ee34febd4082a0253d11b8b47de0"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "6426a75584ec856da11953659c3068da"
server_key_iv: "60912f264aed820d340a81ee39d46ee3"
server_key_version: "encryptionkey@0"
I20260619 12:00:11.218873 7682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:11.235668 7682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:11.238175 7682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:11.238360 7682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:11.238636 7682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:11.238976 7682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:11.239014 7682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.239038 7682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:11.239051 7682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.246990 7682 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:44299
I20260619 12:00:11.247072 7900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:44299 every 8 connection(s)
I20260619 12:00:11.247529 7682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:11.251812 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 7682
I20260619 12:00:11.251991 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 7126
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.252@KRBTEST.COM
I20260619 12:00:11.258527 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:35073
--local_ip_for_outbound_sockets=127.0.203.194
--tserver_master_addrs=127.0.203.254:36661
--webserver_port=41577
--webserver_interface=127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab
--principal=oryx/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:00:11.261351 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.248349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:52023 (local address 127.0.203.254:36661)
0619 12:00:11.248397 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.248400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.248941 (+ 541us) server_negotiation.cc:407] Connection header received
0619 12:00:11.250056 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.250058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.250070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.250139 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.251238 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.251814 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.254162 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.254371 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.254444 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.257798 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.257800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.257800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.257809 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.259543 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.260174 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.260175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.260176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.260238 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.260521 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.260522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.260522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.260692 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.260749 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.261182 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.261272 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:00:11.261682 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.248440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.203.193:45751)
0619 12:00:11.249025 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.249037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.249971 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.250274 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.250276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.250880 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.251111 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.251120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.254033 (+ 2913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.254035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.255132 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.255135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.255634 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.255635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.255715 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.256183 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.256188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.257991 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.260250 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.260252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.260252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.260479 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.260734 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.260734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.260735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.260795 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.261061 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.261062 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.261109 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.261188 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.261389 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":21,"thread_start_us":129,"threads_started":1}
I20260619 12:00:11.262115 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.248349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:45751 (local address 127.0.203.252:32795)
0619 12:00:11.249154 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.249156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.249165 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:11.250122 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.250123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.250140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.250217 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.252864 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.253920 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.255254 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.255525 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.255630 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.258124 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.258127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.258128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.258138 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.260149 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.260559 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.260560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.260561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.260664 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.260856 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.260857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.260857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.260982 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.261037 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.261877 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.261975 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":65,"threads_started":1}
I20260619 12:00:11.262247 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.248405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:52023)
0619 12:00:11.248754 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.248835 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.249932 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.250265 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.250271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.250853 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.251110 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.251116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.251946 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.251948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.254060 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.254063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.254462 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.254463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.254615 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.255469 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.255480 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.257669 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.259646 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.259656 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.259659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.260069 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.260308 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.260309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.260309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.260451 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.260787 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.260793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.260990 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.261464 (+ 474us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.261609 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260619 12:00:11.262498 7902 heartbeater.cc:347] Connected to a master server at 127.0.203.252:32795
I20260619 12:00:11.262594 7902 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.262802 7902 heartbeater.cc:511] Master 127.0.203.252:32795 requested a full tablet report, sending...
I20260619 12:00:11.262817 7901 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36661
I20260619 12:00:11.262869 7901 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.262951 7901 heartbeater.cc:511] Master 127.0.203.254:36661 requested a full tablet report, sending...
I20260619 12:00:11.263302 6930 ts_manager.cc:194] Re-registered known tserver with Master: fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193:44299)
I20260619 12:00:11.263374 7722 ts_manager.cc:194] Registered new tserver with Master: fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193:44299)
I20260619 12:00:11.263808 6930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:52023
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.253@KRBTEST.COM
I20260619 12:00:11.269120 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.248961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:48339 (local address 127.0.203.253:46427)
0619 12:00:11.249024 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.249026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.250263 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 12:00:11.250380 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.250382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.250394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.250459 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.251960 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.252579 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.261166 (+ 8587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.261454 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.261502 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.266114 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.266116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.266117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.266127 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.267841 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.268247 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.268248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.268248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.268306 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.268504 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.268505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.268506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.268676 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.268738 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.268905 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.269012 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:00:11.269146 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.248834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.203.193:48339)
0619 12:00:11.250199 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.250212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.250288 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.250484 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.250485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.250882 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.251111 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.251125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.260104 (+ 8979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.260105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.261046 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.261048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.261656 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.261657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.261723 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.264781 (+ 3058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.264785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.266013 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.267973 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.267974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.267974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.268156 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.268375 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.268376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.268376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.268441 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.268741 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.268743 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.268788 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.268881 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.269016 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260619 12:00:11.269742 7903 heartbeater.cc:347] Connected to a master server at 127.0.203.253:46427
I20260619 12:00:11.269793 7903 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.269878 7903 heartbeater.cc:511] Master 127.0.203.253:46427 requested a full tablet report, sending...
I20260619 12:00:11.270285 7526 ts_manager.cc:194] Registered new tserver with Master: fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193:44299)
W20260619 12:00:11.345427 7909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:11.345718 7909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:11.345762 7909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:11.347100 7909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:11.347175 7909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:11.347223 7909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:11.348966 7909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab
--principal=oryx/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:35073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=41577
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:11.349253 7909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:11.349545 7909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:11.355284 7914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:11.355284 7917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:11.355455 7909 server_base.cc:1034] running on GCE node
W20260619 12:00:11.355613 7915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:11.355865 7909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:11.356135 7909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:11.357281 7909 hybrid_clock.cc:648] HybridClock initialized: now 1781870411357272 us; error 37 us; skew 500 ppm
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:11.360199 7909 init.cc:377] Logged in from keytab as oryx/127.0.203.194@KRBTEST.COM (short username oryx)
I20260619 12:00:11.360816 7909 webserver.cc:460] Webserver started at http://127.0.203.194:41577/ using document root <none> and password file <none>
I20260619 12:00:11.361090 7909 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:11.361155 7909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:11.362789 7909 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:11.363610 7924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.363981 7909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:11.364102 7909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "df0ef59ef3684dc5b7737aaeab902ded"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "3f4081ace1ba237aa5093c9af6a1c9e6"
server_key_iv: "9b95f9efbd376148402ebff058c9be25"
server_key_version: "encryptionkey@0"
I20260619 12:00:11.364570 7909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:11.378118 7909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:11.381505 7909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:11.381843 7909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:11.382192 7909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:11.382826 7909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:11.382893 7909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.382941 7909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:11.382963 7909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.390522 7909 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:35073
I20260619 12:00:11.390621 8037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:35073 every 8 connection(s)
I20260619 12:00:11.391183 7909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:11.395926 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 7909
I20260619 12:00:11.396104 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 7260
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.253@KRBTEST.COM
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.252@KRBTEST.COM
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:11.405275 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:44983
--local_ip_for_outbound_sockets=127.0.203.195
--tserver_master_addrs=127.0.203.254:36661
--webserver_port=44321
--webserver_interface=127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab
--principal=oryx/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:00:11.405666 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.392622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:33183 (local address 127.0.203.253:46427)
0619 12:00:11.392715 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.392718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.393089 (+ 371us) server_negotiation.cc:407] Connection header received
0619 12:00:11.393983 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.393985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.393999 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.394037 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.394857 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.395676 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.397712 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.398026 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.398067 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.401168 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.401169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.401170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.401177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.403995 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.404767 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.404768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.404769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.404836 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.405046 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.405047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.405047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.405175 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.405224 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.405450 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.405552 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:00:11.405665 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.392561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:39721 (local address 127.0.203.252:32795)
0619 12:00:11.392682 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.392686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.392895 (+ 209us) server_negotiation.cc:407] Connection header received
0619 12:00:11.393948 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.393949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.393965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.394037 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.394847 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.395399 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.397247 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.397604 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.397649 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.401633 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.401635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.401636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.401645 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.403508 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.404358 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.404360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.404360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.404456 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.404805 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.404806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.404807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.404981 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.405039 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.405465 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.405568 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:00:11.406376 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.392162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.203.194:33183)
0619 12:00:11.393022 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.393030 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.393862 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.394137 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.394138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.394551 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.394750 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.394759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.395777 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.395778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.397607 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.397609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.398158 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.398159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.398235 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.398794 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.398799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.400997 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.404357 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.404359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.404359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.404674 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.404917 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.404918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.404919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.404973 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.405255 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.405256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.405287 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.405819 (+ 532us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.406144 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":225,"thread_start_us":139,"threads_started":1}
I20260619 12:00:11.406379 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.392164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.203.194:39721)
0619 12:00:11.392749 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.392800 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.393830 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.394115 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.394120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.394505 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.394748 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.394753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.395515 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.395517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.397137 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.397145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.397757 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.397758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.397941 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.398682 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.398693 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.401501 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.403629 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.403638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.403641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.404013 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.404550 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.404551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.404552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.404728 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.405073 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.405080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.405282 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.405819 (+ 537us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.406095 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":3,"spinlock_wait_cycles":8448,"thread_start_us":140,"threads_started":1}
I20260619 12:00:11.406998 8040 heartbeater.cc:347] Connected to a master server at 127.0.203.253:46427
I20260619 12:00:11.407112 8040 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.407351 8039 heartbeater.cc:347] Connected to a master server at 127.0.203.252:32795
I20260619 12:00:11.407358 8040 heartbeater.cc:511] Master 127.0.203.253:46427 requested a full tablet report, sending...
I20260619 12:00:11.407411 8039 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.407461 8039 heartbeater.cc:511] Master 127.0.203.252:32795 requested a full tablet report, sending...
I20260619 12:00:11.407429 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.392512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:45373 (local address 127.0.203.254:36661)
0619 12:00:11.392597 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.392600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.392896 (+ 296us) server_negotiation.cc:407] Connection header received
0619 12:00:11.393994 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.393996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.394009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.394041 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.394855 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.395676 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.397712 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.398023 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.398067 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.402513 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.402515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.402515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.402524 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.404617 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.405093 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.405095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.405096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.405156 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.405628 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.405629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.405629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.405815 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.405874 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.406988 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.407095 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:00:11.407917 7722 ts_manager.cc:194] Registered new tserver with Master: df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194:35073)
I20260619 12:00:11.408404 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.392166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:45373)
0619 12:00:11.392763 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.392806 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.393865 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.394137 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.394138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.394551 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.394750 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.394759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.395777 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.395778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.397607 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.397609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.398156 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.398158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.398781 (+ 623us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.399172 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.399177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.402388 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.404750 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.404751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.404751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.405000 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.405225 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.405225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.405226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.405282 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.406342 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.406343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.406400 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.408074 (+ 1674us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.408251 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":137,"thread_start_us":186,"threads_started":1}
I20260619 12:00:11.409546 7526 ts_manager.cc:194] Registered new tserver with Master: df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194:35073)
I20260619 12:00:11.409971 8038 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36661
I20260619 12:00:11.410022 8038 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.410105 8038 heartbeater.cc:511] Master 127.0.203.254:36661 requested a full tablet report, sending...
I20260619 12:00:11.410425 6930 ts_manager.cc:194] Re-registered known tserver with Master: df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194:35073)
I20260619 12:00:11.411237 6930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:45373
W20260619 12:00:11.499166 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:11.499516 8045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:11.499557 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:11.501049 8045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:11.501096 8045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:11.501124 8045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:11.502632 8045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab
--principal=oryx/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:44983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=44321
--tserver_master_addrs=127.0.203.254:36661,127.0.203.253:46427,127.0.203.252:32795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:11.502848 8045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:11.503049 8045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:11.509187 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:11.509151 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:11.509238 8045 server_base.cc:1034] running on GCE node
W20260619 12:00:11.509178 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:11.509606 8045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:11.509986 8045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:11.511184 8045 hybrid_clock.cc:648] HybridClock initialized: now 1781870411511182 us; error 44 us; skew 500 ppm
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:11.514161 8045 init.cc:377] Logged in from keytab as oryx/127.0.203.195@KRBTEST.COM (short username oryx)
I20260619 12:00:11.514673 8045 webserver.cc:460] Webserver started at http://127.0.203.195:44321/ using document root <none> and password file <none>
I20260619 12:00:11.514946 8045 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:11.515038 8045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:11.516558 8045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:11.517309 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.517555 8045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:11.517670 8045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "932fa8f21d0f48a1b19749a2c7858864"
format_stamp: "Formatted at 2026-06-19 12:00:10 on dist-test-slave-fbq8"
server_key: "5028f9e15cf30acbe061423b5e8c120e"
server_key_iv: "e212cc3cc25d4cbc326ab8086bde13fe"
server_key_version: "encryptionkey@0"
I20260619 12:00:11.518061 8045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:11.538196 8045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:11.540833 8045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:11.541018 8045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:11.541285 8045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:11.541664 8045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:11.541724 8045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.541769 8045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:11.541797 8045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:11.549150 8045 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:44983
I20260619 12:00:11.549299 8173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:44983 every 8 connection(s)
I20260619 12:00:11.549687 8045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:11.554396 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 8045
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.252@KRBTEST.COM
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.253@KRBTEST.COM
Jun 19 12:00:11 dist-test-slave-fbq8 krb5kdc[6885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870411, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:11.565802 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.550634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:58755 (local address 127.0.203.252:32795)
0619 12:00:11.550695 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.550697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.551473 (+ 776us) server_negotiation.cc:407] Connection header received
0619 12:00:11.552492 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.552495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.552511 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.552571 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.553665 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.554279 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.555995 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.556304 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.556369 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.561934 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.561936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.561937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.561946 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.563797 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.564526 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.564528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.564528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.564598 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.564914 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.564915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.564916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.565048 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.565134 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.565516 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.565610 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:00:11.566632 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.550623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.203.195:58755)
0619 12:00:11.551390 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.551399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.552266 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.552703 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.552704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.553189 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.553546 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.553554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.554396 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.554397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.555875 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.555878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.556484 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.556486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.556683 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.557925 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.557930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.561790 (+ 3860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.563905 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.563920 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.563924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.564423 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.564700 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.564702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.564702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.564834 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.565129 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.565135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.565325 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.566090 (+ 765us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.566339 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":596,"thread_start_us":76,"threads_started":1}
I20260619 12:00:11.566814 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.551103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:42063 (local address 127.0.203.253:46427)
0619 12:00:11.551158 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.551161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.551349 (+ 188us) server_negotiation.cc:407] Connection header received
0619 12:00:11.552554 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.552555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.552567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.552630 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.554509 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.555261 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.556540 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.556881 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.556923 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.562151 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.562153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.562154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.562162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.565118 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.565576 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.565578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.565578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.565745 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.566222 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.566223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.566223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.566379 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.566429 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.566640 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.566725 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:00:11.567166 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.551025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.203.195:42063)
0619 12:00:11.551256 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.551270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.552277 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.552687 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.552689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.553189 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.553546 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.553554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.555399 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.555401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.556434 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.556438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.557014 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.557015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.557098 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.557787 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.557794 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.562018 (+ 4224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.565248 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.565249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.565249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.565489 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.566012 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.566013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.566013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.566086 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.566480 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.566481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.566522 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.566892 (+ 370us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.567063 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":27,"thread_start_us":100,"threads_started":1}
I20260619 12:00:11.567366 8175 heartbeater.cc:347] Connected to a master server at 127.0.203.252:32795
I20260619 12:00:11.567457 8175 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.567615 8175 heartbeater.cc:511] Master 127.0.203.252:32795 requested a full tablet report, sending...
I20260619 12:00:11.567888 8176 heartbeater.cc:347] Connected to a master server at 127.0.203.253:46427
I20260619 12:00:11.567929 8176 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.567986 8176 heartbeater.cc:511] Master 127.0.203.253:46427 requested a full tablet report, sending...
I20260619 12:00:11.568149 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.555698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.0.203.253:46427)
0619 12:00:11.555745 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.555747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.556736 (+ 989us) server_negotiation.cc:407] Connection header received
0619 12:00:11.556843 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.556843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.556854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.556884 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.558077 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.558812 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.560057 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.560262 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.560352 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.561914 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.561916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.561916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.561926 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.565492 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.565921 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.565922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.565922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.566772 (+ 850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.567267 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.567268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.567269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.567556 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.567669 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.567944 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.568044 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 12:00:11.568331 7722 ts_manager.cc:194] Registered new tserver with Master: 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983)
I20260619 12:00:11.568349 7526 ts_manager.cc:194] Registered new tserver with Master: 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983)
I20260619 12:00:11.574530 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.550702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:34553 (local address 127.0.203.254:36661)
0619 12:00:11.550754 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.550756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.551053 (+ 297us) server_negotiation.cc:407] Connection header received
0619 12:00:11.552417 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.552419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.552434 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.552465 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.553672 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.554379 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.556002 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.556320 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.556379 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.569089 (+ 12710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.569092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.569092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.570408 (+ 1316us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.572358 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.572936 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.572938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.572938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.572998 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.573222 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.573223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.573223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.573746 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.573974 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.574239 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.574336 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 12:00:11.574954 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.550623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:34553)
0619 12:00:11.550890 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.550931 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.552266 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.552627 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.552632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.553265 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:11.553544 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.553551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.554645 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.554645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.555875 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.555890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.556485 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.556486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.556683 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.557620 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:11.557633 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:11.564223 (+ 6590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:11.572486 (+ 8263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.572488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.572488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.572814 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.573079 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:11.573080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:11.573080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:11.573149 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:11.574028 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:11.574029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:11.574100 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.574457 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.574602 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":13,"thread_start_us":120,"threads_started":1}
I20260619 12:00:11.576606 8174 heartbeater.cc:347] Connected to a master server at 127.0.203.254:36661
I20260619 12:00:11.576745 8174 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:11.576866 8174 heartbeater.cc:511] Master 127.0.203.254:36661 requested a full tablet report, sending...
I20260619 12:00:11.577142 6930 ts_manager.cc:194] Re-registered known tserver with Master: 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983)
I20260619 12:00:11.577716 6930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:34553
I20260619 12:00:11.578677 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.556089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.0.203.254:36661)
0619 12:00:11.558144 (+ 2055us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.558146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.558312 (+ 166us) server_negotiation.cc:407] Connection header received
0619 12:00:11.558344 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.558345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.558362 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.558439 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.559983 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.564716 (+ 4733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.570920 (+ 6204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.571291 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.571384 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.574407 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.574409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.574409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.574418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.576991 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.577381 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.577382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.577383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.577443 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.577656 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.577656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.577657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.577806 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.577908 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.578425 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.578528 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2021,"thread_start_us":75,"threads_started":1}
I20260619 12:00:11.581664 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.556421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.0.203.252:32795)
0619 12:00:11.558350 (+ 1929us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.558352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.558488 (+ 136us) server_negotiation.cc:407] Connection header received
0619 12:00:11.558512 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.558513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.558525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.558575 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.562012 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.563473 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.573462 (+ 9989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.573840 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.573914 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.576442 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.576443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.576444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.576454 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.579860 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.580510 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.580511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.580512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.580593 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.581084 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.581085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.581085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.581247 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.581358 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.581495 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.581582 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1908,"thread_start_us":62,"threads_started":1}
I20260619 12:00:11.580982 6930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:11.584092 6930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:11.596311 8197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.591555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44983 (local address 127.0.0.1:58114)
0619 12:00:11.591874 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.591884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.591927 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.592866 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.592867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.592868 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:11.592908 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.592910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.594055 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.594056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.595048 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.595049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.596026 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.596027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.596093 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.596098 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.596148 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.596198 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":106,"threads_started":1}
I20260619 12:00:11.596577 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.591668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.0.203.195:44983)
0619 12:00:11.592595 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.592599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.592607 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:11.592707 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.592710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.592724 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.592856 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:11.592973 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.593979 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.595169 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.595902 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.596022 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.596093 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.596173 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.596221 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":81,"threads_started":1}
I20260619 12:00:11.597527 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.592024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.0.203.193:44299)
0619 12:00:11.592276 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.592282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.592294 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:11.592337 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.592339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.592357 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.592456 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:11.592772 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.593924 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.596373 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.597146 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.597244 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.597312 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.597405 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.597441 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260619 12:00:11.597520 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.591725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:44299 (local address 127.0.0.1:40968)
0619 12:00:11.591978 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.591989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.592097 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.592498 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.592500 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.592504 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:11.592628 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.592629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.595018 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.595020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.596257 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.596260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.597257 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.597258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.597326 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.597332 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.597391 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.597442 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":38,"thread_start_us":90,"threads_started":1}
I20260619 12:00:11.597616 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.591438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:35073 (local address 127.0.0.1:44390)
0619 12:00:11.591752 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.591764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.591809 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.592517 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.592518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.592519 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:11.592627 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.592629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.594958 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.594959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.595813 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.595815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.597390 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.597391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.597463 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.597467 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.597497 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.597550 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":23}
I20260619 12:00:11.597548 8108 tablet_service.cc:1467] Processing CreateTablet for tablet e0db870710fe47b896b95ff94a19ff56 (DEFAULT_TABLE table=test-table [id=2bfbbf8c0c8e431d9d5d8a07a0f9d134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:11.598076 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0db870710fe47b896b95ff94a19ff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:11.598239 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.591611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.0.203.194:35073)
0619 12:00:11.592276 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.592282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.592294 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:11.592338 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.592340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.592357 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.592456 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:11.594061 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.594869 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.595945 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.597286 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.597979 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.598068 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.598100 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.598156 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":122,"threads_started":1}
I20260619 12:00:11.598362 7835 tablet_service.cc:1467] Processing CreateTablet for tablet e0db870710fe47b896b95ff94a19ff56 (DEFAULT_TABLE table=test-table [id=2bfbbf8c0c8e431d9d5d8a07a0f9d134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:11.598810 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0db870710fe47b896b95ff94a19ff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:11.598918 7972 tablet_service.cc:1467] Processing CreateTablet for tablet e0db870710fe47b896b95ff94a19ff56 (DEFAULT_TABLE table=test-table [id=2bfbbf8c0c8e431d9d5d8a07a0f9d134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:11.599252 7972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0db870710fe47b896b95ff94a19ff56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:11.602375 8199 tablet_bootstrap.cc:492] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Bootstrap starting.
I20260619 12:00:11.603508 8199 tablet_bootstrap.cc:654] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:11.603866 8199 log.cc:824] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:11.604084 8201 tablet_bootstrap.cc:492] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Bootstrap starting.
I20260619 12:00:11.604770 8200 tablet_bootstrap.cc:492] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Bootstrap starting.
I20260619 12:00:11.604833 8201 tablet_bootstrap.cc:654] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:11.604842 8199 tablet_bootstrap.cc:492] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: No bootstrap required, opened a new log
I20260619 12:00:11.604928 8199 ts_tablet_manager.cc:1397] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:00:11.605217 8201 log.cc:824] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:11.605376 8200 tablet_bootstrap.cc:654] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:11.605695 8200 log.cc:824] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:11.606091 8201 tablet_bootstrap.cc:492] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: No bootstrap required, opened a new log
I20260619 12:00:11.606178 8201 ts_tablet_manager.cc:1397] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 12:00:11.606473 8200 tablet_bootstrap.cc:492] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: No bootstrap required, opened a new log
I20260619 12:00:11.606539 8200 ts_tablet_manager.cc:1397] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:00:11.606992 8199 raft_consensus.cc:348] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.607175 8199 raft_consensus.cc:374] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:11.607211 8199 raft_consensus.cc:729] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932fa8f21d0f48a1b19749a2c7858864, State: Initialized, Role: FOLLOWER
I20260619 12:00:11.607333 8199 consensus_queue.cc:260] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.607609 8199 ts_tablet_manager.cc:1428] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:00:11.608283 8174 heartbeater.cc:503] Master 127.0.203.254:36661 was elected leader, sending a full tablet report...
I20260619 12:00:11.609033 8201 raft_consensus.cc:348] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.609179 8201 raft_consensus.cc:374] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:11.609218 8201 raft_consensus.cc:729] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5ee34febd4082a0253d11b8b47de0, State: Initialized, Role: FOLLOWER
I20260619 12:00:11.609436 8201 consensus_queue.cc:260] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.609546 8200 raft_consensus.cc:348] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.609687 8201 ts_tablet_manager.cc:1428] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 12:00:11.609736 8200 raft_consensus.cc:374] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:11.609786 8200 raft_consensus.cc:729] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0ef59ef3684dc5b7737aaeab902ded, State: Initialized, Role: FOLLOWER
I20260619 12:00:11.609895 7901 heartbeater.cc:503] Master 127.0.203.254:36661 was elected leader, sending a full tablet report...
I20260619 12:00:11.609935 8200 consensus_queue.cc:260] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.610191 8200 ts_tablet_manager.cc:1428] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 12:00:11.610317 8038 heartbeater.cc:503] Master 127.0.203.254:36661 was elected leader, sending a full tablet report...
I20260619 12:00:11.611469 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.606402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.0.1:49384)
0619 12:00:11.606499 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.606512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.606556 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.606866 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.606867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.606869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:11.606919 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.606920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.609168 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.609169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.610279 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.610283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.611215 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.611216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.611272 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.611276 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.611325 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.611386 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 12:00:11.611776 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.606642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.0.203.252:32795)
0619 12:00:11.606681 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.606684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.606692 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:11.606736 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.606736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.606760 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.606818 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:11.606984 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.607730 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.610461 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.611039 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.611109 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.611181 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.611643 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.611696 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:00:11.612232 7732 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260619 12:00:11.612785 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a432676ad224681b060d6872c00e574. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.612877 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: LEARNER
W20260619 12:00:11.642225 8041 tablet.cc:2369] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:00:11.748629 7904 tablet.cc:2369] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:00:11.801651 8177 tablet.cc:2369] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:00:11.852478 8206 raft_consensus.cc:482] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:11.852722 8206 raft_consensus.cc:504] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.853250 8206 leader_election.cc:288] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194:35073), 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983)
I20260619 12:00:11.856782 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.853613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:32885 (local address 127.0.203.194:35073)
0619 12:00:11.853752 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.853754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.853773 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:00:11.853848 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.853849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.853861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.853977 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:11.854239 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.854805 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.856005 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.856382 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.856473 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.856482 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.856632 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.856698 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:00:11.856789 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.853892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:36043 (local address 127.0.203.195:44983)
0619 12:00:11.853983 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.853986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.853998 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:11.854032 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.854033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.854044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.854114 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:11.854292 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.854873 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.855999 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.856383 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.856471 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.856483 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.856656 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.856699 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:00:11.856845 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.853511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:35073 (local address 127.0.203.193:32885)
0619 12:00:11.853678 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.853693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.853771 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.854046 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.854047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.854050 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:11.854115 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.854117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.854887 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.854889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.855912 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.855914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.856472 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.856473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.856551 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.856555 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.856625 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.856718 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":28,"thread_start_us":108,"threads_started":1}
I20260619 12:00:11.856877 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.853754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44983 (local address 127.0.203.193:36043)
0619 12:00:11.853891 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:11.853900 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:11.853972 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:11.854119 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:11.854120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:11.854123 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:11.854215 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.854217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.854992 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.854993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.855914 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:11.855916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.856477 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.856478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:11.856557 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.856561 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:11.856628 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:00:11.856699 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260619 12:00:11.857348 7992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0db870710fe47b896b95ff94a19ff56" candidate_uuid: "fac5ee34febd4082a0253d11b8b47de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0ef59ef3684dc5b7737aaeab902ded" is_pre_election: true
I20260619 12:00:11.857568 7992 raft_consensus.cc:2454] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5ee34febd4082a0253d11b8b47de0 in term 0.
I20260619 12:00:11.857766 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0db870710fe47b896b95ff94a19ff56" candidate_uuid: "fac5ee34febd4082a0253d11b8b47de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932fa8f21d0f48a1b19749a2c7858864" is_pre_election: true
I20260619 12:00:11.857860 7788 leader_election.cc:302] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df0ef59ef3684dc5b7737aaeab902ded, fac5ee34febd4082a0253d11b8b47de0; no voters:
I20260619 12:00:11.857973 8128 raft_consensus.cc:2454] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac5ee34febd4082a0253d11b8b47de0 in term 0.
I20260619 12:00:11.858031 8206 raft_consensus.cc:2781] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:11.858098 8206 raft_consensus.cc:482] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:11.858142 8206 raft_consensus.cc:3037] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:11.858918 8206 raft_consensus.cc:504] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.859072 8206 leader_election.cc:288] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [CANDIDATE]: Term 1 election: Requested vote from peers df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194:35073), 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983)
I20260619 12:00:11.859308 7992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0db870710fe47b896b95ff94a19ff56" candidate_uuid: "fac5ee34febd4082a0253d11b8b47de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0ef59ef3684dc5b7737aaeab902ded"
I20260619 12:00:11.859414 7992 raft_consensus.cc:3037] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:11.859313 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0db870710fe47b896b95ff94a19ff56" candidate_uuid: "fac5ee34febd4082a0253d11b8b47de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932fa8f21d0f48a1b19749a2c7858864"
I20260619 12:00:11.859483 8128 raft_consensus.cc:3037] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:11.860158 7992 raft_consensus.cc:2454] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5ee34febd4082a0253d11b8b47de0 in term 1.
I20260619 12:00:11.860345 7788 leader_election.cc:302] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df0ef59ef3684dc5b7737aaeab902ded, fac5ee34febd4082a0253d11b8b47de0; no voters:
I20260619 12:00:11.860469 8206 raft_consensus.cc:2781] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:11.860536 8128 raft_consensus.cc:2454] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac5ee34febd4082a0253d11b8b47de0 in term 1.
I20260619 12:00:11.860612 8206 raft_consensus.cc:686] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 1 LEADER]: Becoming Leader. State: Replica: fac5ee34febd4082a0253d11b8b47de0, State: Running, Role: LEADER
I20260619 12:00:11.860738 8206 consensus_queue.cc:237] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:11.861864 6930 catalog_manager.cc:5495] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "fac5ee34febd4082a0253d11b8b47de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } health_report { overall_health: HEALTHY } } }
I20260619 12:00:11.862624 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: NON_VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: true } } }
I20260619 12:00:11.862746 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.863602 7686 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: attempting to promote NON_VOTER 271f9c8643584928b89cff5a7a9447c7 to VOTER
I20260619 12:00:11.863749 7686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
I20260619 12:00:11.864360 7538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Refusing update from remote peer 4a432676ad224681b060d6872c00e574: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 12:00:11.864625 7732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 LEARNER]: Refusing update from remote peer 4a432676ad224681b060d6872c00e574: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 12:00:11.864706 8216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 12:00:11.864867 8216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260619 12:00:11.865514 7489 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.866010 7732 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.866612 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.866645 8216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.866699 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.866740 8217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.866801 8217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.866706 8216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:11.867203 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.867305 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:00:11.867231 7538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.868187 8219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a432676ad224681b060d6872c00e574" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:11.868280 8219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:00:11.885819 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.883182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.0.203.193:44299)
0619 12:00:11.883236 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.883238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.883248 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:11.883286 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.883286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.883305 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.883385 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:11.883507 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.884046 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.885056 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.885249 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.885318 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.885503 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:11.885611 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:11.885722 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.885738 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.885758 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:00:11.887964 8128 raft_consensus.cc:1261] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Refusing update from remote peer fac5ee34febd4082a0253d11b8b47de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:11.887984 7992 raft_consensus.cc:1261] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 1 FOLLOWER]: Refusing update from remote peer fac5ee34febd4082a0253d11b8b47de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:11.888494 8214 consensus_queue.cc:1035] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:11.888659 8206 consensus_queue.cc:1035] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:11.893829 8222 mvcc.cc:205] Tried to move back new op lower bound from 7298541207089508352 to 7298541206982299648. Current Snapshot: MvccSnapshot[applied={T|T < 7298541207089508352}]
I20260619 12:00:11.896111 6930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:11.896203 6930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:11.897903 6930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a432676ad224681b060d6872c00e574: Sending DeleteTablet for 3 replicas of tablet e0db870710fe47b896b95ff94a19ff56
I20260619 12:00:11.898269 7972 tablet_service.cc:1514] Processing DeleteTablet for tablet e0db870710fe47b896b95ff94a19ff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:11 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44390
I20260619 12:00:11.898320 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet e0db870710fe47b896b95ff94a19ff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:11 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:58114
I20260619 12:00:11.898792 8231 tablet_replica.cc:330] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: stopping tablet replica
I20260619 12:00:11.898779 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet e0db870710fe47b896b95ff94a19ff56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:11 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:40968
I20260619 12:00:11.898911 8231 raft_consensus.cc:2229] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:11.899068 8231 raft_consensus.cc:2258] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:11.899273 8234 tablet_replica.cc:330] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: stopping tablet replica
I20260619 12:00:11.899386 8234 raft_consensus.cc:2229] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:00:11.899511 8234 raft_consensus.cc:2258] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:11.899688 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 6898
I20260619 12:00:11.901708 8231 ts_tablet_manager.cc:1905] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:11.903636 8231 ts_tablet_manager.cc:1918] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:11.903729 8231 log.cc:1196] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/e0db870710fe47b896b95ff94a19ff56
I20260619 12:00:11.904064 8231 ts_tablet_manager.cc:1939] T e0db870710fe47b896b95ff94a19ff56 P df0ef59ef3684dc5b7737aaeab902ded: Deleting consensus metadata
W20260619 12:00:11.907034 7927 connection.cc:535] server connection from 127.0.0.1:44390 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:00:11.907696 8232 tablet_replica.cc:330] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: stopping tablet replica
I20260619 12:00:11.907881 8232 raft_consensus.cc:2229] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:11.907943 8234 ts_tablet_manager.cc:1905] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:11.908026 8232 raft_consensus.cc:2258] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:11.908675 8232 ts_tablet_manager.cc:1905] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:11.910091 8234 ts_tablet_manager.cc:1918] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:11.910172 8234 log.cc:1196] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/e0db870710fe47b896b95ff94a19ff56
I20260619 12:00:11.910504 8234 ts_tablet_manager.cc:1939] T e0db870710fe47b896b95ff94a19ff56 P fac5ee34febd4082a0253d11b8b47de0: Deleting consensus metadata
I20260619 12:00:11.910527 8232 ts_tablet_manager.cc:1918] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:11.910607 8232 log.cc:1196] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/e0db870710fe47b896b95ff94a19ff56
I20260619 12:00:11.910905 8232 ts_tablet_manager.cc:1939] T e0db870710fe47b896b95ff94a19ff56 P 932fa8f21d0f48a1b19749a2c7858864: Deleting consensus metadata
W20260619 12:00:11.911309 7789 connection.cc:441] server connection from 127.0.0.1:40968 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40968 (request call id 1) could send its response
W20260619 12:00:11.911334 8062 connection.cc:441] server connection from 127.0.0.1:58114 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58114 (request call id 1) could send its response
I20260619 12:00:11.914705 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.899526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.0.203.252:32795)
0619 12:00:11.899570 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.899573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.899805 (+ 232us) server_negotiation.cc:407] Connection header received
0619 12:00:11.901295 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.901297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.901310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.901351 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.904393 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.905203 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.906362 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.906655 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.906701 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.909067 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.909070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.909070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.909081 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.911162 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.913130 (+ 1968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.913131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.913132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.913195 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.913398 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.913399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.913399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.913922 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.914034 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.914474 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.914590 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 12:00:11.921208 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.911498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.0.203.252:32795)
0619 12:00:11.911559 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.911565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.912043 (+ 478us) server_negotiation.cc:407] Connection header received
0619 12:00:11.912150 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.912151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.912163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.912223 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.913010 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.914284 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.915729 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.915966 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.916045 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.917806 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.917808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.917808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.917816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.919558 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.920021 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.920023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.920023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.920105 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.920357 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.920358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.920358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.920503 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.920586 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.920857 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.920965 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 12:00:11.921479 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:11.911999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.0.203.253:46427)
0619 12:00:11.912062 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:11.912065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:11.912292 (+ 227us) server_negotiation.cc:407] Connection header received
0619 12:00:11.912374 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:11.912375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:11.912387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:11.912489 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:11.913640 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.914536 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.915897 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:11.916202 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:11.916242 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:11.918072 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:11.918073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:11.918073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:11.918081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:11.919741 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.920146 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.920148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.920148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.920226 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:11.920434 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:11.920435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:11.920436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:11.920591 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:11.920841 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:11.921261 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 12:00:11.921371 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 12:00:12.893819 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:12.893325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:49661)
0619 12:00:12.893569 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:12.893723 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
W20260619 12:00:12.894163 8174 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:36661 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:12.894678 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:12.894356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:38615)
0619 12:00:12.894522 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:12.894622 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260619 12:00:12.895155 7901 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:36661 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:12.895443 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:12.895168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:41439)
0619 12:00:12.895305 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:12.895387 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
W20260619 12:00:12.895642 8038 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:36661 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:13.398434 8244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a432676ad224681b060d6872c00e574)
I20260619 12:00:13.398624 8244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
I20260619 12:00:13.399019 8244 leader_election.cc:288] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661), 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252:32795)
I20260619 12:00:13.399037 8245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a432676ad224681b060d6872c00e574)
I20260619 12:00:13.399479 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.399200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34514)
0619 12:00:13.399374 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.399432 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260619 12:00:13.399220 8245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
W20260619 12:00:13.399619 7513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:13.400000 8245 leader_election.cc:288] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661), 0c06179b2a2c4090b9bed40befcfaebd (127.0.203.253:46427)
I20260619 12:00:13.400504 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.399876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34522)
0619 12:00:13.400003 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.400457 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260619 12:00:13.400533 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.400441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34530)
0619 12:00:13.400482 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.400506 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 12:00:13.400599 7513 leader_election.cc:334] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661): Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
W20260619 12:00:13.400709 7707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:13.401229 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.401063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34532)
0619 12:00:13.401176 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.401203 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
W20260619 12:00:13.401297 7707 leader_election.cc:334] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661): Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:13.403203 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.399414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.0.203.252:32795)
0619 12:00:13.399760 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:13.399764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:13.399777 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:00:13.399816 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:13.399817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:13.399831 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:13.399957 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:13.400249 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.401057 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.402394 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.402796 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.402900 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:13.402914 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:13.403089 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 12:00:13.403133 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":59,"threads_started":1}
I20260619 12:00:13.403208 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.399244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:32795 (local address 127.0.0.1:49406)
0619 12:00:13.399485 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.399498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:13.399622 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:13.400043 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:13.400045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:13.400048 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:13.400126 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:13.400128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.401166 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.401167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:13.402282 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:13.402284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.402904 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.402905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:13.403003 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:13.403008 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:13.403082 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:00:13.403142 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":29,"thread_start_us":65,"threads_started":1}
I20260619 12:00:13.403457 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.399919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:46427 (local address 127.0.0.1:39642)
0619 12:00:13.400171 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.400183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:13.400238 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:13.400416 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:13.400417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:13.400421 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:13.400485 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:13.400488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.401811 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.401812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:13.402596 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:13.402597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.403197 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.403198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:13.403265 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:13.403270 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:13.403326 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 12:00:13.403390 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":45,"threads_started":1}
I20260619 12:00:13.403445 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c06179b2a2c4090b9bed40befcfaebd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "271f9c8643584928b89cff5a7a9447c7" is_pre_election: true
I20260619 12:00:13.403605 7732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c06179b2a2c4090b9bed40befcfaebd in term 1.
I20260619 12:00:13.403626 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.399987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.0.203.253:46427)
0619 12:00:13.400257 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:13.400259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:13.400274 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:00:13.400303 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:13.400304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:13.400315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:13.400393 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:13.401026 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.401714 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.402700 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:13.403113 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:13.403199 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:13.403213 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:13.403485 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 12:00:13.403535 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
I20260619 12:00:13.403769 7512 leader_election.cc:302] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c06179b2a2c4090b9bed40befcfaebd, 271f9c8643584928b89cff5a7a9447c7; no voters: 4a432676ad224681b060d6872c00e574
I20260619 12:00:13.403836 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "271f9c8643584928b89cff5a7a9447c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0c06179b2a2c4090b9bed40befcfaebd" is_pre_election: true
I20260619 12:00:13.403914 8244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 12:00:13.403964 7538 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271f9c8643584928b89cff5a7a9447c7 in term 1.
I20260619 12:00:13.403990 8244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a432676ad224681b060d6872c00e574)
I20260619 12:00:13.404022 8244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:00:13.404147 7708 leader_election.cc:302] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c06179b2a2c4090b9bed40befcfaebd, 271f9c8643584928b89cff5a7a9447c7; no voters: 4a432676ad224681b060d6872c00e574
I20260619 12:00:13.404273 8245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 12:00:13.404313 8245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a432676ad224681b060d6872c00e574)
I20260619 12:00:13.404333 8245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:00:13.405021 8244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
I20260619 12:00:13.405198 8244 leader_election.cc:288] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 2 election: Requested vote from peers 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661), 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252:32795)
I20260619 12:00:13.405390 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.405287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34544)
0619 12:00:13.405334 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.405361 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 12:00:13.405329 8245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
I20260619 12:00:13.405391 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c06179b2a2c4090b9bed40befcfaebd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "271f9c8643584928b89cff5a7a9447c7"
I20260619 12:00:13.405458 7732 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c06179b2a2c4090b9bed40befcfaebd in current term 2: Already voted for candidate 271f9c8643584928b89cff5a7a9447c7 in this term.
I20260619 12:00:13.405495 8245 leader_election.cc:288] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [CANDIDATE]: Term 2 election: Requested vote from peers 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661), 0c06179b2a2c4090b9bed40befcfaebd (127.0.203.253:46427)
I20260619 12:00:13.405669 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "271f9c8643584928b89cff5a7a9447c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0c06179b2a2c4090b9bed40befcfaebd"
I20260619 12:00:13.405751 7538 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 271f9c8643584928b89cff5a7a9447c7 in current term 2: Already voted for candidate 0c06179b2a2c4090b9bed40befcfaebd in this term.
I20260619 12:00:13.405988 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.405821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34558)
0619 12:00:13.405930 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.405966 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 12:00:13.405992 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.405588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34546)
0619 12:00:13.405943 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.405970 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
W20260619 12:00:13.406148 7513 leader_election.cc:334] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661): Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:13.406214 7513 leader_election.cc:302] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c06179b2a2c4090b9bed40befcfaebd; no voters: 271f9c8643584928b89cff5a7a9447c7, 4a432676ad224681b060d6872c00e574
I20260619 12:00:13.406335 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.406253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:34572)
0619 12:00:13.406290 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.406316 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 12:00:13.406389 8244 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260619 12:00:13.406406 7707 leader_election.cc:334] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661): Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:13.406451 7707 leader_election.cc:302] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 271f9c8643584928b89cff5a7a9447c7; no voters: 0c06179b2a2c4090b9bed40befcfaebd, 4a432676ad224681b060d6872c00e574
I20260619 12:00:13.406531 8245 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 12:00:13.895041 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.894707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:60067)
0619 12:00:13.894936 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.894994 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":138,"threads_started":1}
I20260619 12:00:13.895923 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.895547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:55829)
0619 12:00:13.895791 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.895847 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":139,"threads_started":1}
I20260619 12:00:13.896232 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:13.895955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:47963)
0619 12:00:13.896112 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:13.896182 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260619 12:00:14.895888 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.895609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:52993)
0619 12:00:14.895757 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.895825 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260619 12:00:14.896909 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.896554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:40475)
0619 12:00:14.896761 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.896829 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260619 12:00:14.896973 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.896789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:36909)
0619 12:00:14.896888 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.896923 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":46,"threads_started":1}
I20260619 12:00:14.912515 8261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:14.912632 8261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
I20260619 12:00:14.912951 8261 leader_election.cc:288] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661), 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252:32795)
I20260619 12:00:14.913246 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c06179b2a2c4090b9bed40befcfaebd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "271f9c8643584928b89cff5a7a9447c7" is_pre_election: true
I20260619 12:00:14.913357 7732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c06179b2a2c4090b9bed40befcfaebd in term 2.
I20260619 12:00:14.913378 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.913119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:52640)
0619 12:00:14.913268 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.913320 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260619 12:00:14.913524 7512 leader_election.cc:302] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c06179b2a2c4090b9bed40befcfaebd, 271f9c8643584928b89cff5a7a9447c7; no voters:
I20260619 12:00:14.913645 8261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 12:00:14.913694 8261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:14.913719 8261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:00:14.913975 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.913871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:52650)
0619 12:00:14.913923 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.913949 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 12:00:14.914105 7513 leader_election.cc:334] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661): Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:14.914644 8261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
I20260619 12:00:14.914892 8261 leader_election.cc:288] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 3 election: Requested vote from peers 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661), 271f9c8643584928b89cff5a7a9447c7 (127.0.203.252:32795)
I20260619 12:00:14.915045 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.914902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:52652)
0619 12:00:14.914974 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.915012 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:00:14.915086 7732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c06179b2a2c4090b9bed40befcfaebd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "271f9c8643584928b89cff5a7a9447c7"
I20260619 12:00:14.915182 7732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:00:14.915496 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.915352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:52664)
0619 12:00:14.915421 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.915463 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:00:14.915604 7513 leader_election.cc:334] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661): Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
I20260619 12:00:14.916054 7732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c06179b2a2c4090b9bed40befcfaebd in term 3.
I20260619 12:00:14.916296 7512 leader_election.cc:302] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c06179b2a2c4090b9bed40befcfaebd, 271f9c8643584928b89cff5a7a9447c7; no voters: 4a432676ad224681b060d6872c00e574
I20260619 12:00:14.916409 8261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:00:14.916555 8261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [term 3 LEADER]: Becoming Leader. State: Replica: 0c06179b2a2c4090b9bed40befcfaebd, State: Running, Role: LEADER
I20260619 12:00:14.916657 8261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } }
I20260619 12:00:14.917038 8264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c06179b2a2c4090b9bed40befcfaebd. Latest consensus state: current_term: 3 leader_uuid: "0c06179b2a2c4090b9bed40befcfaebd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:14.917140 8264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:14.917361 8266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:14.917912 8266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:14.917982 8266 catalog_manager.cc:1259] Loaded cluster ID: c75bda7ddbe2420f95092867a6a000a5
I20260619 12:00:14.918040 8266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:14.918257 8266 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:14.918349 8266 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Loaded TSK: 0
I20260619 12:00:14.918526 8266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:00:14.938537 7527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:14.940315 7527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:14.941581 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.941431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:52684)
0619 12:00:14.941490 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.941524 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 12:00:14.941684 7732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [term 3 FOLLOWER]: Refusing update from remote peer 0c06179b2a2c4090b9bed40befcfaebd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 12:00:14.942001 8264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [LEADER]: Connected to new peer: Peer: permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 12:00:14.942137 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.941993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.0.1:52698)
0619 12:00:14.942053 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.942087 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 12:00:14.942239 7513 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd -> Peer 4a432676ad224681b060d6872c00e574 (127.0.203.254:36661): Couldn't send request to peer 4a432676ad224681b060d6872c00e574. Status: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 12:00:14.942725 8268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c06179b2a2c4090b9bed40befcfaebd. Latest consensus state: current_term: 3 leader_uuid: "0c06179b2a2c4090b9bed40befcfaebd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:14.942875 8268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:00:14.943521 8261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0c06179b2a2c4090b9bed40befcfaebd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:14.943651 8261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:14.943871 8268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0c06179b2a2c4090b9bed40befcfaebd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4a432676ad224681b060d6872c00e574" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36661 } } peers { permanent_uuid: "0c06179b2a2c4090b9bed40befcfaebd" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 46427 } attrs { promote: false } } peers { permanent_uuid: "271f9c8643584928b89cff5a7a9447c7" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 32795 } attrs { promote: false } } }
I20260619 12:00:14.943958 8268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271f9c8643584928b89cff5a7a9447c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:00:14.950277 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.946604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.0.203.194:35073)
0619 12:00:14.946904 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:14.946907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:14.946919 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:14.946979 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:14.946980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:14.946992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:14.947130 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:14.947347 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.948266 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.949409 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.949887 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.949963 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:14.949979 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:14.950138 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 12:00:14.950189 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":72,"threads_started":1}
I20260619 12:00:14.950279 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.946532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:35073 (local address 127.0.0.1:54634)
0619 12:00:14.946790 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.946805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:14.946881 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:14.947115 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:14.947117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:14.947119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:14.947198 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:14.947200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.948361 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.948362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:14.949313 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:14.949315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.949985 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.949986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:14.950053 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:14.950058 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:14.950119 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:00:14.950177 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":1}
I20260619 12:00:14.950882 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.946777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.0.203.195:44983)
0619 12:00:14.946973 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:14.946976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:14.947046 (+ 70us) server_negotiation.cc:407] Connection header received
0619 12:00:14.947132 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:14.947133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:14.947144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:14.947210 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:14.947352 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.948146 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.949722 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.950320 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.950386 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:14.950401 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:14.950723 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 12:00:14.950767 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260619 12:00:14.950666 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.946630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44983 (local address 127.0.0.1:47254)
0619 12:00:14.946982 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.946995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:14.947065 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:14.947210 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:14.947211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:14.947213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:14.947273 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:14.947275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.948245 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.948246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:14.949613 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:14.949616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.950438 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.950439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:14.950494 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:14.950497 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:14.950538 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 12:00:14.950583 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":67,"thread_start_us":115,"threads_started":1}
I20260619 12:00:14.951118 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.946642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:44299 (local address 127.0.0.1:55466)
0619 12:00:14.947204 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.947215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:14.947327 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:14.947508 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:14.947508 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:14.947510 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:14.947556 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:14.947557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.948690 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.948691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:14.949777 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:14.949778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.950823 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.950824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:14.950909 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:14.950916 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:00:14.950972 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 12:00:14.951044 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":120,"threads_started":1}
I20260619 12:00:14.951097 7972 tablet_service.cc:1467] Processing CreateTablet for tablet 913e2c612f13484d8a905b3004c7fef3 (DEFAULT_TABLE table=test-table [id=d083f27d822a4b38955dc1eba49daec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:14.951203 8108 tablet_service.cc:1467] Processing CreateTablet for tablet 913e2c612f13484d8a905b3004c7fef3 (DEFAULT_TABLE table=test-table [id=d083f27d822a4b38955dc1eba49daec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:14.951290 7972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913e2c612f13484d8a905b3004c7fef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:14.951454 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.946767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.203.193:44299)
0619 12:00:14.946924 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:14.946926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:14.947301 (+ 375us) server_negotiation.cc:407] Connection header received
0619 12:00:14.947397 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:14.947398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:14.947410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:14.947507 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:14.947630 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.948562 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.949879 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:14.950709 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:14.950817 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:14.950834 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:14.951297 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 12:00:14.951348 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260619 12:00:14.951483 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913e2c612f13484d8a905b3004c7fef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:14.952796 7835 tablet_service.cc:1467] Processing CreateTablet for tablet 913e2c612f13484d8a905b3004c7fef3 (DEFAULT_TABLE table=test-table [id=d083f27d822a4b38955dc1eba49daec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:14.952945 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913e2c612f13484d8a905b3004c7fef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:14.953168 8280 tablet_bootstrap.cc:492] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: Bootstrap starting.
I20260619 12:00:14.953577 8280 tablet_bootstrap.cc:654] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:14.954341 8280 tablet_bootstrap.cc:492] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: No bootstrap required, opened a new log
I20260619 12:00:14.954409 8280 ts_tablet_manager.cc:1397] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:14.954556 8280 raft_consensus.cc:348] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:14.954654 8280 raft_consensus.cc:374] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:14.954679 8280 raft_consensus.cc:729] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932fa8f21d0f48a1b19749a2c7858864, State: Initialized, Role: FOLLOWER
I20260619 12:00:14.954768 8279 tablet_bootstrap.cc:492] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: Bootstrap starting.
I20260619 12:00:14.954814 8280 consensus_queue.cc:260] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:14.954893 8282 tablet_bootstrap.cc:492] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0: Bootstrap starting.
I20260619 12:00:14.955022 8280 ts_tablet_manager.cc:1428] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:14.955202 8279 tablet_bootstrap.cc:654] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:14.955387 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.955269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:42081)
0619 12:00:14.955322 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.955351 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 12:00:14.955404 8282 tablet_bootstrap.cc:654] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:14.955925 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.955775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:53155)
0619 12:00:14.955848 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.955889 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:00:14.956405 8176 heartbeater.cc:503] Master 127.0.203.253:46427 was elected leader, sending a full tablet report...
I20260619 12:00:14.956758 8279 tablet_bootstrap.cc:492] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: No bootstrap required, opened a new log
I20260619 12:00:14.956816 8279 ts_tablet_manager.cc:1397] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:00:14.956985 8279 raft_consensus.cc:348] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:14.957083 8279 raft_consensus.cc:374] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:14.957108 8279 raft_consensus.cc:729] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0ef59ef3684dc5b7737aaeab902ded, State: Initialized, Role: FOLLOWER
I20260619 12:00:14.957178 8282 tablet_bootstrap.cc:492] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0: No bootstrap required, opened a new log
I20260619 12:00:14.957162 8279 consensus_queue.cc:260] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:14.957237 8282 ts_tablet_manager.cc:1397] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:00:14.957409 8279 ts_tablet_manager.cc:1428] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:14.957401 8282 raft_consensus.cc:348] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:14.957475 8282 raft_consensus.cc:374] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:14.957500 8282 raft_consensus.cc:729] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac5ee34febd4082a0253d11b8b47de0, State: Initialized, Role: FOLLOWER
I20260619 12:00:14.957554 8282 consensus_queue.cc:260] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:14.957768 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.957577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:56809)
0619 12:00:14.957699 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.957731 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 12:00:14.957744 8282 ts_tablet_manager.cc:1428] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:14.958039 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.957962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:55287)
0619 12:00:14.957993 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.958013 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 12:00:14.958352 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.958252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:35137)
0619 12:00:14.958297 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.958319 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 12:00:14.958536 8040 heartbeater.cc:503] Master 127.0.203.253:46427 was elected leader, sending a full tablet report...
I20260619 12:00:14.958705 7903 heartbeater.cc:503] Master 127.0.203.253:46427 was elected leader, sending a full tablet report...
I20260619 12:00:14.958809 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:14.958733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:44803)
0619 12:00:14.958763 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:14.958786 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260619 12:00:15.113960 8286 raft_consensus.cc:482] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:15.114079 8286 raft_consensus.cc:504] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:15.114424 8286 leader_election.cc:288] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983), fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193:44299)
I20260619 12:00:15.117691 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.114649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:44299 (local address 127.0.203.194:33933)
0619 12:00:15.114748 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.114765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:15.114841 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:15.115057 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:15.115059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:15.115063 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:15.115164 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:15.115166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.115856 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.115857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:15.116789 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:15.116792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.117379 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.117380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:15.117461 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:15.117466 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:15.117549 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:00:15.117598 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260619 12:00:15.117676 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.114814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:33933 (local address 127.0.203.193:44299)
0619 12:00:15.114908 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:15.114910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:15.114919 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:15.114946 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:15.114947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:15.114957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:15.115040 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:15.115242 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.115752 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.116889 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.117271 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.117385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:15.117396 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:15.117558 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 12:00:15.117597 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:00:15.118018 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913e2c612f13484d8a905b3004c7fef3" candidate_uuid: "df0ef59ef3684dc5b7737aaeab902ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac5ee34febd4082a0253d11b8b47de0" is_pre_election: true
I20260619 12:00:15.118259 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.114670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44983 (local address 127.0.203.194:43751)
0619 12:00:15.115103 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.115113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:15.115437 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:15.115645 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:15.115646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:15.115648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:15.115742 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:15.115744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.116417 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.116418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:15.117262 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:15.117264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.117972 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.117973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:15.118055 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:15.118059 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:15.118120 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:00:15.118164 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":244,"thread_start_us":137,"threads_started":1}
I20260619 12:00:15.118256 7855 raft_consensus.cc:2454] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0ef59ef3684dc5b7737aaeab902ded in term 0.
I20260619 12:00:15.118270 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.114733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:43751 (local address 127.0.203.195:44983)
0619 12:00:15.114820 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:15.114822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:15.115412 (+ 590us) server_negotiation.cc:407] Connection header received
0619 12:00:15.115538 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:15.115539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:15.115550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:15.115638 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:15.115813 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.116318 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.117420 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.117889 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.117965 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:15.117976 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:15.118146 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 12:00:15.118194 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:00:15.118482 7927 leader_election.cc:302] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df0ef59ef3684dc5b7737aaeab902ded, fac5ee34febd4082a0253d11b8b47de0; no voters:
I20260619 12:00:15.118690 8286 raft_consensus.cc:2781] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:15.118786 8286 raft_consensus.cc:482] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:15.118816 8286 raft_consensus.cc:3037] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:15.118789 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913e2c612f13484d8a905b3004c7fef3" candidate_uuid: "df0ef59ef3684dc5b7737aaeab902ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932fa8f21d0f48a1b19749a2c7858864" is_pre_election: true
I20260619 12:00:15.118889 8128 raft_consensus.cc:2454] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0ef59ef3684dc5b7737aaeab902ded in term 0.
I20260619 12:00:15.119801 8286 raft_consensus.cc:504] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:15.119988 8286 leader_election.cc:288] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [CANDIDATE]: Term 1 election: Requested vote from peers 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983), fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193:44299)
I20260619 12:00:15.120182 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913e2c612f13484d8a905b3004c7fef3" candidate_uuid: "df0ef59ef3684dc5b7737aaeab902ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932fa8f21d0f48a1b19749a2c7858864"
I20260619 12:00:15.120198 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913e2c612f13484d8a905b3004c7fef3" candidate_uuid: "df0ef59ef3684dc5b7737aaeab902ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac5ee34febd4082a0253d11b8b47de0"
I20260619 12:00:15.120265 7855 raft_consensus.cc:3037] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:15.120265 8128 raft_consensus.cc:3037] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:15.121220 8128 raft_consensus.cc:2454] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0ef59ef3684dc5b7737aaeab902ded in term 1.
I20260619 12:00:15.121304 7855 raft_consensus.cc:2454] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0ef59ef3684dc5b7737aaeab902ded in term 1.
I20260619 12:00:15.121470 7927 leader_election.cc:302] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df0ef59ef3684dc5b7737aaeab902ded, fac5ee34febd4082a0253d11b8b47de0; no voters:
I20260619 12:00:15.121680 8286 raft_consensus.cc:2781] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:15.121954 8286 raft_consensus.cc:686] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 1 LEADER]: Becoming Leader. State: Replica: df0ef59ef3684dc5b7737aaeab902ded, State: Running, Role: LEADER
I20260619 12:00:15.122061 8286 consensus_queue.cc:237] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } }
I20260619 12:00:15.122447 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.122281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:47351)
0619 12:00:15.122364 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.122408 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 12:00:15.122999 7527 catalog_manager.cc:5495] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded reported cstate change: term changed from 0 to 1, leader changed from <none> to df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194). New cstate: current_term: 1 leader_uuid: "df0ef59ef3684dc5b7737aaeab902ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ef59ef3684dc5b7737aaeab902ded" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 35073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 } health_report { overall_health: UNKNOWN } } }
I20260619 12:00:15.134244 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.131668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.0.203.194:35073)
0619 12:00:15.131753 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:15.131755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:15.131767 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:15.131807 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:15.131808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:15.131851 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:15.131941 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:15.132089 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.132618 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.133509 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:15.133700 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:15.133765 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:15.133908 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:15.134030 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:15.134095 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:15.134155 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 12:00:15.134175 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:00:15.135926 8128 raft_consensus.cc:1261] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Refusing update from remote peer df0ef59ef3684dc5b7737aaeab902ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:15.135998 7855 raft_consensus.cc:1261] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 1 FOLLOWER]: Refusing update from remote peer df0ef59ef3684dc5b7737aaeab902ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:15.136353 8286 consensus_queue.cc:1035] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "932fa8f21d0f48a1b19749a2c7858864" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:15.136605 8286 consensus_queue.cc:1035] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac5ee34febd4082a0253d11b8b47de0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:15.137436 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.137284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:44465)
0619 12:00:15.137338 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.137378 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 12:00:15.138022 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.137891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:54225)
0619 12:00:15.137961 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.137988 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 12:00:15.139570 8295 mvcc.cc:205] Tried to move back new op lower bound from 7298541220393689088 to 7298541220340916224. Current Snapshot: MvccSnapshot[applied={T|T < 7298541220393689088 or (T in {7298541220393689088})}]
I20260619 12:00:15.139643 8293 mvcc.cc:205] Tried to move back new op lower bound from 7298541220393689088 to 7298541220340916224. Current Snapshot: MvccSnapshot[applied={T|T < 7298541220393689088}]
I20260619 12:00:15.139954 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.139204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:53691)
0619 12:00:15.139888 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.139922 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663}
I20260619 12:00:15.140275 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.140174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.195:49187)
0619 12:00:15.140217 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.140241 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 12:00:15.140053 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.139851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.193:45273)
0619 12:00:15.140003 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.140029 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260619 12:00:15.140391 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.140307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:36661 (local address 127.0.203.194:34959)
0619 12:00:15.140346 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.140367 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 12:00:15.143213 7526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:15.143313 7526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:15.145028 7526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c06179b2a2c4090b9bed40befcfaebd: Sending DeleteTablet for 3 replicas of tablet 913e2c612f13484d8a905b3004c7fef3
I20260619 12:00:15.145447 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet 913e2c612f13484d8a905b3004c7fef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:15 UTC) from {username='oryx', principal='oryx/127.0.203.253@KRBTEST.COM'} at 127.0.0.1:55466
I20260619 12:00:15.145733 8302 tablet_replica.cc:330] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0: stopping tablet replica
I20260619 12:00:15.145846 7972 tablet_service.cc:1514] Processing DeleteTablet for tablet 913e2c612f13484d8a905b3004c7fef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:15 UTC) from {username='oryx', principal='oryx/127.0.203.253@KRBTEST.COM'} at 127.0.0.1:54634
I20260619 12:00:15.145874 8302 raft_consensus.cc:2229] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:15.145973 8302 raft_consensus.cc:2258] T 913e2c612f13484d8a905b3004c7fef3 P fac5ee34febd4082a0253d11b8b47de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:15.145956 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet 913e2c612f13484d8a905b3004c7fef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:15 UTC) from {username='oryx', principal='oryx/127.0.203.253@KRBTEST.COM'} at 127.0.0.1:47254
I20260619 12:00:15.146183 8304 tablet_replica.cc:330] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: stopping tablet replica
I20260619 12:00:15.146298 8304 raft_consensus.cc:2229] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:15.146394 8304 raft_consensus.cc:2258] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:15.146620 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 7682
I20260619 12:00:15.147439 8304 ts_tablet_manager.cc:1905] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:15.149391 8304 ts_tablet_manager.cc:1918] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:15.149478 8304 log.cc:1196] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/913e2c612f13484d8a905b3004c7fef3
I20260619 12:00:15.149745 8303 tablet_replica.cc:330] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: stopping tablet replica
I20260619 12:00:15.149871 8303 raft_consensus.cc:2229] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:00:15.149912 8304 ts_tablet_manager.cc:1939] T 913e2c612f13484d8a905b3004c7fef3 P 932fa8f21d0f48a1b19749a2c7858864: Deleting consensus metadata
I20260619 12:00:15.150095 8303 raft_consensus.cc:2258] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:15.150345 7512 catalog_manager.cc:4841] TS 932fa8f21d0f48a1b19749a2c7858864 (127.0.203.195:44983): tablet 913e2c612f13484d8a905b3004c7fef3 (table test-table [id=d083f27d822a4b38955dc1eba49daec7]) successfully deleted
I20260619 12:00:15.150714 8303 ts_tablet_manager.cc:1905] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:15.153045 8303 ts_tablet_manager.cc:1918] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:15.153115 8303 log.cc:1196] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/913e2c612f13484d8a905b3004c7fef3
I20260619 12:00:15.153354 8303 ts_tablet_manager.cc:1939] T 913e2c612f13484d8a905b3004c7fef3 P df0ef59ef3684dc5b7737aaeab902ded: Deleting consensus metadata
I20260619 12:00:15.153972 7512 catalog_manager.cc:4841] TS df0ef59ef3684dc5b7737aaeab902ded (127.0.203.194:35073): tablet 913e2c612f13484d8a905b3004c7fef3 (table test-table [id=d083f27d822a4b38955dc1eba49daec7]) successfully deleted
I20260619 12:00:15.156802 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:15.156676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:44299 (local address 127.0.0.1:55480)
0619 12:00:15.156724 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:15.156754 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 12:00:15.157100 7514 catalog_manager.cc:4584] TS fac5ee34febd4082a0253d11b8b47de0 (127.0.203.193:44299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 913e2c612f13484d8a905b3004c7fef3: Network error: Client connection negotiation failed: client connection to 127.0.203.193:44299: connect: Connection refused (error 111)
I20260619 12:00:15.157335 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 7909
I20260619 12:00:15.163321 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 8045
I20260619 12:00:15.169313 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 7394
I20260619 12:00:15.176060 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 7582
2026-06-19T12:00:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6648 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:15 dist-test-slave-fbq8 krb5kdc[8307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:15 dist-test-slave-fbq8 krb5kdc[8307](info): set up 2 sockets
Jun 19 12:00:15 dist-test-slave-fbq8 krb5kdc[8307](info): commencing operation
krb5kdc: starting...
W20260619 12:00:16.249308 815 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:16 dist-test-slave-fbq8 krb5kdc[8307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab.
Entry for principal oryx/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab.
I20260619 12:00:16.380463 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:39449
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab
--principal=oryx/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:16.480948 8320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:16.481201 8320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:16.481228 8320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:16.482576 8320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:16.482622 8320 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 12:00:16.482636 8320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:16.482650 8320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:16.482663 8320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:16.482717 8320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:16.484239 8320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:39449
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.254.keytab
--principal=oryx/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:16.484529 8320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:16.484822 8320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:16.490792 8328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:16.491086 8320 server_base.cc:1034] running on GCE node
W20260619 12:00:16.490814 8326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:16.490885 8325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:16.491575 8320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:16.491914 8320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:16.493100 8320 hybrid_clock.cc:648] HybridClock initialized: now 1781870416493073 us; error 50 us; skew 500 ppm
Jun 19 12:00:16 dist-test-slave-fbq8 krb5kdc[8307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:16.496001 8320 init.cc:377] Logged in from keytab as oryx/127.0.203.254@KRBTEST.COM (short username oryx)
I20260619 12:00:16.496536 8320 webserver.cc:460] Webserver started at http://127.0.203.254:40849/ using document root <none> and password file <none>
I20260619 12:00:16.496824 8320 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:16.496879 8320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:16.497025 8320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:16.498248 8320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "b18b68b05110476eaea7509f70dd29a6"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "c4dfd221c5f1f02a9f77e6c486618c3b"
server_key_iv: "1a53da6c87819d0f203743fe83b34f5f"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.498667 8320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "b18b68b05110476eaea7509f70dd29a6"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "c4dfd221c5f1f02a9f77e6c486618c3b"
server_key_iv: "1a53da6c87819d0f203743fe83b34f5f"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.500249 8320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:16.501343 8335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:16.501657 8320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:00:16.501832 8320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "b18b68b05110476eaea7509f70dd29a6"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "c4dfd221c5f1f02a9f77e6c486618c3b"
server_key_iv: "1a53da6c87819d0f203743fe83b34f5f"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.501953 8320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:16.517028 8320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:16.519618 8320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:16.519827 8320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:16.525345 8320 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:39449
I20260619 12:00:16.525401 8397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:39449 every 8 connection(s)
I20260619 12:00:16.525756 8320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:16.526528 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:16.527788 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 8320
I20260619 12:00:16.527874 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:16.528107 815 external_mini_cluster.cc:1428] Setting key eef5f80befdbda00b55dcceeac4ba611
I20260619 12:00:16.529901 8398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Bootstrap starting.
I20260619 12:00:16.530642 8398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:16.530997 8398 log.cc:824] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:16.531790 8398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: No bootstrap required, opened a new log
I20260619 12:00:16.534008 8398 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b68b05110476eaea7509f70dd29a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39449 } }
I20260619 12:00:16.534217 8398 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:16.534269 8398 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18b68b05110476eaea7509f70dd29a6, State: Initialized, Role: FOLLOWER
I20260619 12:00:16.534482 8398 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b68b05110476eaea7509f70dd29a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39449 } }
I20260619 12:00:16.534590 8398 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:16.534636 8398 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:16.534687 8398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:16.535540 8398 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b68b05110476eaea7509f70dd29a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39449 } }
I20260619 12:00:16.535671 8398 leader_election.cc:302] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18b68b05110476eaea7509f70dd29a6; no voters:
I20260619 12:00:16.535895 8398 leader_election.cc:288] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 19 12:00:16 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:16.536010 8403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:16.536271 8398 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:16.536283 8403 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [term 1 LEADER]: Becoming Leader. State: Replica: b18b68b05110476eaea7509f70dd29a6, State: Running, Role: LEADER
I20260619 12:00:16.536538 8403 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b68b05110476eaea7509f70dd29a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39449 } }
I20260619 12:00:16.536999 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18b68b05110476eaea7509f70dd29a6. Latest consensus state: current_term: 1 leader_uuid: "b18b68b05110476eaea7509f70dd29a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b68b05110476eaea7509f70dd29a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39449 } } }
I20260619 12:00:16.537256 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:16.537173 8405 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18b68b05110476eaea7509f70dd29a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b68b05110476eaea7509f70dd29a6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39449 } } }
I20260619 12:00:16.537578 8405 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:16.537704 8412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:16.538342 8412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:16.541429 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.528795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.0.203.254:39449)
0619 12:00:16.529125 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:16.529137 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:16.529181 (+ 44us) server_negotiation.cc:407] Connection header received
0619 12:00:16.529828 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:16.529836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:16.529892 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:16.530062 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:16.530781 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.531800 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.532561 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.533543 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.533674 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.536841 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:16.536847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:16.536847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:16.536864 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:16.539597 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.540079 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.540083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.540084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.540195 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.540415 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.540416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.540417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.540744 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:16.540912 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:16.541017 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 12:00:16.541175 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":135,"threads_started":1}
I20260619 12:00:16.541822 8412 catalog_manager.cc:1347] Generated new cluster ID: 288302e243364fc6842a878fbeb0a888
I20260619 12:00:16.541929 8412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:16.578486 8412 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:16.579134 8412 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:16.586380 8412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Generated new TSK 0
I20260619 12:00:16.586597 8412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab.
Entry for principal oryx/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab.
I20260619 12:00:16.623788 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39449
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab
--principal=oryx/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:16.711453 8424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:16.711722 8424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:16.711768 8424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:16.713037 8424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:16.713107 8424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:16.713145 8424 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:00:16.713196 8424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:16.714872 8424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab
--principal=oryx/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:39449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:16.715168 8424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:16.715423 8424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:16.721381 8432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:16.721366 8430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:16.721482 8429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:16.721858 8424 server_base.cc:1034] running on GCE node
I20260619 12:00:16.722163 8424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:16.722461 8424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:16.723639 8424 hybrid_clock.cc:648] HybridClock initialized: now 1781870416723633 us; error 74 us; skew 500 ppm
Jun 19 12:00:16 dist-test-slave-fbq8 krb5kdc[8307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:16.726894 8424 init.cc:377] Logged in from keytab as oryx/127.0.203.193@KRBTEST.COM (short username oryx)
I20260619 12:00:16.727397 8424 webserver.cc:460] Webserver started at http://127.0.203.193:35661/ using document root <none> and password file <none>
I20260619 12:00:16.727602 8424 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:16.727653 8424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:16.727747 8424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:16.729197 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "a7d0d7d11415485c804e2e3268ab737b"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "795f3ca68aceb5a6b1791109bb94a680"
server_key_iv: "51f3507bb47deeae892472c3a3fde5ca"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.729689 8424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "a7d0d7d11415485c804e2e3268ab737b"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "795f3ca68aceb5a6b1791109bb94a680"
server_key_iv: "51f3507bb47deeae892472c3a3fde5ca"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.731271 8424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:16.732216 8439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:16.732560 8424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:16.732731 8424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "a7d0d7d11415485c804e2e3268ab737b"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "795f3ca68aceb5a6b1791109bb94a680"
server_key_iv: "51f3507bb47deeae892472c3a3fde5ca"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.732822 8424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:16.754416 8424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:16.756798 8424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:16.756985 8424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:16.757658 8424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:16.757717 8424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:16.757764 8424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:16.757800 8424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:16.765812 8424 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:46565
I20260619 12:00:16.765894 8554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:46565 every 8 connection(s)
Jun 19 12:00:16 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:16.766311 8424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:16.770262 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.758470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:51961 (local address 127.0.203.254:39449)
0619 12:00:16.758541 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:16.758543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:16.758815 (+ 272us) server_negotiation.cc:407] Connection header received
0619 12:00:16.759494 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:16.759495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:16.759507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:16.759557 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:16.760680 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.761575 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.763284 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.763642 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.763709 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.766890 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:16.766892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:16.766893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:16.766901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:16.768369 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.768937 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.768938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.768938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.769008 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.769362 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.769364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.769365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.769536 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:16.769615 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:16.770000 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 12:00:16.770138 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 12:00:16.770298 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 8424
I20260619 12:00:16.770298 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.758418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39449 (local address 127.0.203.193:51961)
0619 12:00:16.758666 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:16.758713 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:16.759368 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:16.759670 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:16.759675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:16.760144 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:16.760490 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.760499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.761694 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.761696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.763159 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.763164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.763714 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.763716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.763873 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.764680 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:16.764690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:16.766774 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:16.768469 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.768476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.768478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.768854 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.769088 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.769089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.769089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.769218 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.769692 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:16.769697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:16.769831 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 12:00:16.769936 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 12:00:16.770084 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260619 12:00:16.770411 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:16.770675 815 external_mini_cluster.cc:1428] Setting key 5375168ca0e49f8c9b533b2391be8caa
I20260619 12:00:16.771137 8555 heartbeater.cc:347] Connected to a master server at 127.0.203.254:39449
I20260619 12:00:16.771255 8555 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:16.771495 8555 heartbeater.cc:511] Master 127.0.203.254:39449 requested a full tablet report, sending...
I20260619 12:00:16.772162 8351 ts_manager.cc:194] Registered new tserver with Master: a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565)
I20260619 12:00:16.773088 8351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:51961
I20260619 12:00:16.779670 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.772719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:46491 (local address 127.0.203.254:39449)
0619 12:00:16.772765 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:16.772767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:16.772807 (+ 40us) server_negotiation.cc:407] Connection header received
0619 12:00:16.772894 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:16.772894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:16.772904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:16.772968 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:16.773443 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.774234 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.775229 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.775427 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.775505 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.776735 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:16.776736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:16.776737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:16.776745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:16.778400 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.778843 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.778844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.778844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.778908 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.779151 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.779152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.779152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.779279 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:16.779337 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:16.779536 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 12:00:16.779602 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 12:00:16.779765 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.772508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39449 (local address 127.0.203.193:46491)
0619 12:00:16.772725 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:16.772739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:16.772820 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:16.772976 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:16.772977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:16.773216 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:16.773303 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.773305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.774364 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.774365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.775122 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.775125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.775531 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.775532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.775622 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.776094 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:16.776100 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:16.776598 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:16.778503 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.778505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.778505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.778743 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.778992 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.778993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.778993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.779075 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.779356 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:16.779357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:16.779412 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 12:00:16.779517 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 12:00:16.779636 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab.
Entry for principal oryx/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab.
I20260619 12:00:16.799765 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39449
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab
--principal=oryx/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:16.890422 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:16.890702 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:16.890738 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:16.891901 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:16.891984 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:16.892019 8564 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:00:16.892086 8564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:16.893687 8564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab
--principal=oryx/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:39449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:16.894018 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:16.894279 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:16.900435 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:16.900436 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:16.900563 8564 server_base.cc:1034] running on GCE node
W20260619 12:00:16.900436 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:16.901126 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:16.901461 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:16.902668 8564 hybrid_clock.cc:648] HybridClock initialized: now 1781870416902697 us; error 130 us; skew 500 ppm
Jun 19 12:00:16 dist-test-slave-fbq8 krb5kdc[8307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:16.905990 8564 init.cc:377] Logged in from keytab as oryx/127.0.203.194@KRBTEST.COM (short username oryx)
I20260619 12:00:16.906534 8564 webserver.cc:460] Webserver started at http://127.0.203.194:36333/ using document root <none> and password file <none>
I20260619 12:00:16.906787 8564 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:16.906841 8564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:16.906945 8564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:16.908089 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "850ce0b5cea94c68bc86936bfefb2008"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "dcdaedb3eec31a1b08ea8e20bcf9c187"
server_key_iv: "d156757173b4ef65e187f26be7117d4c"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.908444 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "850ce0b5cea94c68bc86936bfefb2008"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "dcdaedb3eec31a1b08ea8e20bcf9c187"
server_key_iv: "d156757173b4ef65e187f26be7117d4c"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.910040 8564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:16.911051 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:16.911331 8564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:16.911446 8564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "850ce0b5cea94c68bc86936bfefb2008"
format_stamp: "Formatted at 2026-06-19 12:00:16 on dist-test-slave-fbq8"
server_key: "dcdaedb3eec31a1b08ea8e20bcf9c187"
server_key_iv: "d156757173b4ef65e187f26be7117d4c"
server_key_version: "encryptionkey@0"
I20260619 12:00:16.911535 8564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:16.933179 8564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:16.935529 8564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:16.935714 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:16.936409 8564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:16.936453 8564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:16.936481 8564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:16.936494 8564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:00:16 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:16.945951 8564 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:37505
I20260619 12:00:16.946378 8564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:16.946461 8694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:37505 every 8 connection(s)
I20260619 12:00:16.948791 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.937286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:60843 (local address 127.0.203.254:39449)
0619 12:00:16.937341 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:16.937343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:16.937578 (+ 235us) server_negotiation.cc:407] Connection header received
0619 12:00:16.938252 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:16.938253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:16.938265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:16.938315 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:16.939416 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.940320 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.941958 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.942313 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.942410 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.945306 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:16.945307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:16.945308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:16.945316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:16.947226 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.947725 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.947727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.947727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.947787 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.948022 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.948023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.948023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.948145 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:16.948200 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:16.948517 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 12:00:16.948608 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 12:00:16.948868 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.937252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39449 (local address 127.0.203.194:60843)
0619 12:00:16.937444 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:16.937487 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:16.938140 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:16.938449 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:16.938453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:16.939031 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:16.939277 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.939282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.940474 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.940475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.941821 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.941826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.942452 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.942453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.942618 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.943331 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:16.943344 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:16.945207 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:16.947315 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.947321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.947323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.947578 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.947856 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.947857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.947857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.947957 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.948206 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:16.948210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:16.948349 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 12:00:16.948444 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:00:16.948576 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
I20260619 12:00:16.949543 8695 heartbeater.cc:347] Connected to a master server at 127.0.203.254:39449
I20260619 12:00:16.949643 8695 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:16.949849 8695 heartbeater.cc:511] Master 127.0.203.254:39449 requested a full tablet report, sending...
I20260619 12:00:16.950376 8352 ts_manager.cc:194] Registered new tserver with Master: 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505)
I20260619 12:00:16.951025 8352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:60843
I20260619 12:00:16.956383 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 8564
I20260619 12:00:16.956518 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:16.956810 815 external_mini_cluster.cc:1428] Setting key f6f0c799c4e9303122c0a40a96d3ebad
I20260619 12:00:16.957440 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.950798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39449 (local address 127.0.203.194:49785)
0619 12:00:16.950957 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:16.950977 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:16.951061 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:16.951248 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:16.951248 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:16.951477 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:16.951552 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.951554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.952285 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.952286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.952906 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:16.952908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.953330 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.953331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:16.953401 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.953974 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:16.953979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:16.954312 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:16.955981 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.955982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.955982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.956287 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.956608 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:16.956609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:16.956610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:16.956719 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:16.957130 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:16.957131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:16.957191 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 12:00:16.957261 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:00:16.957354 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260619 12:00:16.957547 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:16.950850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:49785 (local address 127.0.203.254:39449)
0619 12:00:16.950898 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:16.950900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:16.951043 (+ 143us) server_negotiation.cc:407] Connection header received
0619 12:00:16.951132 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:16.951133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:16.951144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:16.951196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:16.951677 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.952172 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.953003 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:16.953192 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:16.953284 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:16.954416 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:16.954417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:16.954417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:16.954424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:16.955888 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.956401 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.956402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.956402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.956453 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:16.956787 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:16.956787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:16.956788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:16.957010 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:16.957114 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:16.957296 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 12:00:16.957375 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab.
Entry for principal oryx/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab.
I20260619 12:00:16.988814 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39449
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab
--principal=oryx/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:17.076200 8704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:17.076459 8704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:17.076485 8704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:17.077701 8704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:17.077756 8704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:17.077777 8704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:00:17.077796 8704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:17.079308 8704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab
--principal=oryx/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.203.254:39449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:17.079566 8704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:17.079828 8704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:17.085803 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:17.085960 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:17.085815 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:17.086094 8704 server_base.cc:1034] running on GCE node
I20260619 12:00:17.086774 8704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:17.087141 8704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:17.088367 8704 hybrid_clock.cc:648] HybridClock initialized: now 1781870417088354 us; error 33 us; skew 500 ppm
Jun 19 12:00:17 dist-test-slave-fbq8 krb5kdc[8307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:17.091535 8704 init.cc:377] Logged in from keytab as oryx/127.0.203.195@KRBTEST.COM (short username oryx)
I20260619 12:00:17.092077 8704 webserver.cc:460] Webserver started at http://127.0.203.195:34289/ using document root <none> and password file <none>
I20260619 12:00:17.092332 8704 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:17.092404 8704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:17.092568 8704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:17.093804 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "4c13ac479b9b43aa8f8a659d418462a4"
format_stamp: "Formatted at 2026-06-19 12:00:17 on dist-test-slave-fbq8"
server_key: "bdbad35e78f506ecb7e0ea18d44a2dce"
server_key_iv: "31d29ba9bac19780721f5ad50d14c0d8"
server_key_version: "encryptionkey@0"
I20260619 12:00:17.094209 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "4c13ac479b9b43aa8f8a659d418462a4"
format_stamp: "Formatted at 2026-06-19 12:00:17 on dist-test-slave-fbq8"
server_key: "bdbad35e78f506ecb7e0ea18d44a2dce"
server_key_iv: "31d29ba9bac19780721f5ad50d14c0d8"
server_key_version: "encryptionkey@0"
I20260619 12:00:17.095861 8704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:17.097110 8719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:17.097553 8704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:17.097675 8704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "4c13ac479b9b43aa8f8a659d418462a4"
format_stamp: "Formatted at 2026-06-19 12:00:17 on dist-test-slave-fbq8"
server_key: "bdbad35e78f506ecb7e0ea18d44a2dce"
server_key_iv: "31d29ba9bac19780721f5ad50d14c0d8"
server_key_version: "encryptionkey@0"
I20260619 12:00:17.097775 8704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:17.125777 8704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:17.128419 8704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:17.128605 8704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:17.129364 8704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:17.129469 8704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:17.129513 8704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:17.129535 8704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:00:17 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:17.139299 8704 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:45215
I20260619 12:00:17.139423 8834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:45215 every 8 connection(s)
I20260619 12:00:17.139830 8704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:17.142534 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.130006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:35259 (local address 127.0.203.254:39449)
0619 12:00:17.130109 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.130112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.130291 (+ 179us) server_negotiation.cc:407] Connection header received
0619 12:00:17.130854 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.130856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.130872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.130983 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:17.131933 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.132772 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.134832 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.135144 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.135224 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.139044 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:17.139046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:17.139047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:17.139057 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:17.141012 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.141579 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.141580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.141580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.141684 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.141915 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.141915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.141916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.142029 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:17.142080 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.142374 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.142465 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:00:17.142620 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.129952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39449 (local address 127.0.203.195:35259)
0619 12:00:17.130136 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.130180 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.130730 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.131073 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.131079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.131491 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:17.131828 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.131834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.132913 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.132916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.134672 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.134680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.135272 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.135273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.135505 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.136376 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:17.136414 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:17.138877 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:17.141129 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:17.141135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:17.141137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:17.141504 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:17.141761 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:17.141761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:17.141762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:17.141865 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:17.142082 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:17.142085 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:17.142214 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.142305 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.142447 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260619 12:00:17.143368 8835 heartbeater.cc:347] Connected to a master server at 127.0.203.254:39449
I20260619 12:00:17.143461 8835 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:17.143682 8835 heartbeater.cc:511] Master 127.0.203.254:39449 requested a full tablet report, sending...
I20260619 12:00:17.144191 8352 ts_manager.cc:194] Registered new tserver with Master: 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215)
I20260619 12:00:17.144712 8352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:35259
I20260619 12:00:17.145555 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 8704
I20260619 12:00:17.145654 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:17.145828 815 external_mini_cluster.cc:1428] Setting key 9790f97452df2cc69dcac032fe6007e4
I20260619 12:00:17.146865 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:00:17.151991 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.144614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39449 (local address 127.0.203.195:43943)
0619 12:00:17.144780 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.144791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.144885 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.145023 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.145024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.145245 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:17.145319 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.145322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.146027 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.146029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.146836 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.146842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.147304 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.147305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.147390 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.147907 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:17.147913 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:17.148552 (+ 639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:17.150546 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:17.150548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:17.150549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:17.150845 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:17.151120 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:17.151121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:17.151122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:17.151220 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:17.151596 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:17.151597 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:17.151695 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.151782 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.151903 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260619 12:00:17.152084 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.144714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:43943 (local address 127.0.203.254:39449)
0619 12:00:17.144774 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.144776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.144872 (+ 96us) server_negotiation.cc:407] Connection header received
0619 12:00:17.144947 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.144948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.144959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.145016 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:17.145399 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.145924 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.146962 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.147187 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.147247 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.148809 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:17.148810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:17.148810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:17.148817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:17.150448 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.150947 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.150949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.150950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.151022 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.151312 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.151313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.151314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.151484 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:17.151563 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.151829 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.151938 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 12:00:17 dist-test-slave-fbq8 krb5kdc[8307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870417, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:00:17 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:17 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:00:17.173493 8401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:17.167234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.203.254:39449)
0619 12:00:17.167336 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.167338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.167351 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:00:17.167383 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.167385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.167397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.167509 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:17.168192 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.168851 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.169615 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.169800 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.169860 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.173424 (+ 3564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51850: BlockingRecv error: recv got EOF from 127.0.0.1:51850 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 12:00:17 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870417, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:17.182258 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.174548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.0.203.254:39449)
0619 12:00:17.174620 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.174622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.174717 (+ 95us) server_negotiation.cc:407] Connection header received
0619 12:00:17.174789 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.174790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.174802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.174855 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:17.175360 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.175862 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.176599 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.176858 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.176942 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.179337 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:17.179339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:17.179339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:17.179345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:17.180908 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.181312 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.181313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.181314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.181373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.181667 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.181668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.181668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.181793 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:17.181894 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.182090 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.182184 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:00:17.184262 8352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:17.186465 8352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:17.194363 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.189679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:45215 (local address 127.0.0.1:53406)
0619 12:00:17.189933 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.189982 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.190053 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.190334 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.190338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.190341 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:17.190420 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.190423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.191702 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.191703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.193199 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.193201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.194015 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.194016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.194097 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.194114 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.194185 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.194272 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":10,"thread_start_us":128,"threads_started":1}
I20260619 12:00:17.194538 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.189895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.0.203.194:37505)
0619 12:00:17.190119 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.190124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.190364 (+ 240us) server_negotiation.cc:407] Connection header received
0619 12:00:17.190499 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.190501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.190519 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.190583 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:17.190835 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.191837 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.193299 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.194067 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.194133 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.194206 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.194322 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.194365 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260619 12:00:17.194619 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.189942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.203.195:45215)
0619 12:00:17.190129 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.190135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.190145 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:17.190188 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.190190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.190202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.190326 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:17.190509 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.191599 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.193332 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.193901 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.194014 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.194112 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.194419 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.194507 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260619 12:00:17.194748 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.189680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37505 (local address 127.0.0.1:44064)
0619 12:00:17.190266 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.190276 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.190397 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.190675 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.190676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.190680 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:17.190737 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.190738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.191937 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.191937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.193195 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.193198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.194165 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.194166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.194228 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.194231 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.194290 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.194331 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":47,"spinlock_wait_cycles":896,"thread_start_us":129,"threads_started":1}
I20260619 12:00:17.195612 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.189713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:46565 (local address 127.0.0.1:44196)
0619 12:00:17.190416 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.190428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.190482 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.190698 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.190699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.190701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:17.190736 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.190738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.192882 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.192883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.193978 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.193981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.195336 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.195338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.195411 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.195416 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.195486 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.195535 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":116,"threads_started":1}
I20260619 12:00:17.195662 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.190035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.0.203.193:46565)
0619 12:00:17.190209 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.190213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.190554 (+ 341us) server_negotiation.cc:407] Connection header received
0619 12:00:17.190603 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.190605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.190621 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.190700 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:17.191926 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.192798 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.194086 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.195243 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.195319 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.195384 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.195509 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.195550 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260619 12:00:17.195640 8769 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3c1d8993454325ae27bb6db4e47a9c (DEFAULT_TABLE table=test-table [id=f0e317c05ac0435ab87f9bd20b4238a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:17.195881 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3c1d8993454325ae27bb6db4e47a9c (DEFAULT_TABLE table=test-table [id=f0e317c05ac0435ab87f9bd20b4238a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:17.196046 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3c1d8993454325ae27bb6db4e47a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.196296 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3c1d8993454325ae27bb6db4e47a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.196511 8488 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3c1d8993454325ae27bb6db4e47a9c (DEFAULT_TABLE table=test-table [id=f0e317c05ac0435ab87f9bd20b4238a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:17.196878 8488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3c1d8993454325ae27bb6db4e47a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.199694 8863 tablet_bootstrap.cc:492] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Bootstrap starting.
I20260619 12:00:17.200459 8864 tablet_bootstrap.cc:492] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Bootstrap starting.
I20260619 12:00:17.200549 8863 tablet_bootstrap.cc:654] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.200917 8865 tablet_bootstrap.cc:492] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Bootstrap starting.
I20260619 12:00:17.200987 8863 log.cc:824] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:17.201248 8864 tablet_bootstrap.cc:654] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.201630 8864 log.cc:824] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:17.201758 8865 tablet_bootstrap.cc:654] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.202119 8863 tablet_bootstrap.cc:492] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: No bootstrap required, opened a new log
I20260619 12:00:17.202211 8863 ts_tablet_manager.cc:1397] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:00:17.202245 8865 log.cc:824] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:17.202504 8864 tablet_bootstrap.cc:492] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: No bootstrap required, opened a new log
I20260619 12:00:17.202595 8864 ts_tablet_manager.cc:1397] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:00:17.203176 8865 tablet_bootstrap.cc:492] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: No bootstrap required, opened a new log
I20260619 12:00:17.203308 8865 ts_tablet_manager.cc:1397] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:00:17.204217 8864 raft_consensus.cc:348] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.204419 8864 raft_consensus.cc:374] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.204468 8864 raft_consensus.cc:729] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.204615 8864 consensus_queue.cc:260] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.204577 8863 raft_consensus.cc:348] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.204808 8863 raft_consensus.cc:374] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.204850 8863 raft_consensus.cc:729] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850ce0b5cea94c68bc86936bfefb2008, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.204937 8864 ts_tablet_manager.cc:1428] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 12:00:17.204983 8863 consensus_queue.cc:260] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.205117 8835 heartbeater.cc:503] Master 127.0.203.254:39449 was elected leader, sending a full tablet report...
I20260619 12:00:17.205265 8863 ts_tablet_manager.cc:1428] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:00:17.205410 8695 heartbeater.cc:503] Master 127.0.203.254:39449 was elected leader, sending a full tablet report...
I20260619 12:00:17.205669 8865 raft_consensus.cc:348] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.205814 8865 raft_consensus.cc:374] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.205845 8865 raft_consensus.cc:729] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d0d7d11415485c804e2e3268ab737b, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.205947 8865 consensus_queue.cc:260] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.206219 8865 ts_tablet_manager.cc:1428] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:00:17.206272 8555 heartbeater.cc:503] Master 127.0.203.254:39449 was elected leader, sending a full tablet report...
W20260619 12:00:17.267208 8556 tablet.cc:2369] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:00:17.390870 8836 tablet.cc:2369] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:00:17.449836 8869 raft_consensus.cc:482] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:17.450003 8869 raft_consensus.cc:504] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.450564 8869 leader_election.cc:288] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505), a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565)
W20260619 12:00:17.450544 8696 tablet.cc:2369] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:00:17.453902 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.450905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:53683 (local address 127.0.203.194:37505)
0619 12:00:17.450970 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.450973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.451203 (+ 230us) server_negotiation.cc:407] Connection header received
0619 12:00:17.451285 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.451286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.451296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.451363 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:17.451539 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.452029 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.453159 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.453516 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.453597 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.453606 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.453785 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.453822 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 12:00:17.453938 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.450814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37505 (local address 127.0.203.195:53683)
0619 12:00:17.451110 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.451130 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.451225 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.451369 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.451370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.451382 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:17.451468 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.451471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.452117 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.452118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.453070 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.453072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.453606 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.453607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.453699 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.453702 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.453778 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.453832 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":53,"thread_start_us":144,"threads_started":1}
I20260619 12:00:17.453945 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.451057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:39353 (local address 127.0.203.193:46565)
0619 12:00:17.451139 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.451141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.451150 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:17.451179 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.451180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.451189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.451263 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:17.451449 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.451980 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.453155 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.453534 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.453619 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.453630 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.453817 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.453860 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:00:17.454319 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.450808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:46565 (local address 127.0.203.195:39353)
0619 12:00:17.450868 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.450883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.450942 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.451268 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.451270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.451272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:17.451358 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.451361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.452064 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.452065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.453044 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.453047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.453625 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.453626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.453731 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.453734 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.453809 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.453862 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":62}
I20260619 12:00:17.454562 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3c1d8993454325ae27bb6db4e47a9c" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008" is_pre_election: true
I20260619 12:00:17.454749 8649 raft_consensus.cc:2454] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 0.
I20260619 12:00:17.454785 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3c1d8993454325ae27bb6db4e47a9c" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b" is_pre_election: true
I20260619 12:00:17.454973 8499 raft_consensus.cc:2454] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 0.
I20260619 12:00:17.455052 8723 leader_election.cc:302] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c13ac479b9b43aa8f8a659d418462a4, 850ce0b5cea94c68bc86936bfefb2008; no voters:
I20260619 12:00:17.455173 8869 raft_consensus.cc:2781] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:17.455226 8869 raft_consensus.cc:482] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:17.455255 8869 raft_consensus.cc:3037] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:17.456382 8869 raft_consensus.cc:504] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.456557 8869 leader_election.cc:288] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 election: Requested vote from peers 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505), a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565)
I20260619 12:00:17.456739 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3c1d8993454325ae27bb6db4e47a9c" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008"
I20260619 12:00:17.456856 8649 raft_consensus.cc:3037] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:17.456838 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3c1d8993454325ae27bb6db4e47a9c" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b"
I20260619 12:00:17.456935 8499 raft_consensus.cc:3037] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:17.457865 8499 raft_consensus.cc:2454] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 1.
I20260619 12:00:17.457871 8649 raft_consensus.cc:2454] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 1.
I20260619 12:00:17.458052 8723 leader_election.cc:302] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c13ac479b9b43aa8f8a659d418462a4, 850ce0b5cea94c68bc86936bfefb2008; no voters:
I20260619 12:00:17.458153 8869 raft_consensus.cc:2781] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:17.458340 8869 raft_consensus.cc:686] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Running, Role: LEADER
I20260619 12:00:17.458464 8869 consensus_queue.cc:237] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } }
I20260619 12:00:17.459532 8351 catalog_manager.cc:5495] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195). New cstate: current_term: 1 leader_uuid: "4c13ac479b9b43aa8f8a659d418462a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } health_report { overall_health: UNKNOWN } } }
I20260619 12:00:17.483946 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.481258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.0.203.195:45215)
0619 12:00:17.481372 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.481374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.481384 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:17.481419 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.481420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.481437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.481510 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:17.481690 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.482314 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.483125 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.483310 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.483368 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.483531 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:17.483664 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:17.483733 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.483801 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.483822 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 12:00:17.486146 8649 raft_consensus.cc:1261] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Refusing update from remote peer 4c13ac479b9b43aa8f8a659d418462a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:17.486181 8499 raft_consensus.cc:1261] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Refusing update from remote peer 4c13ac479b9b43aa8f8a659d418462a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:17.486555 8873 consensus_queue.cc:1035] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:17.486721 8869 consensus_queue.cc:1035] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:17.489526 8876 mvcc.cc:205] Tried to move back new op lower bound from 7298541230020136960 to 7298541229910614016. Current Snapshot: MvccSnapshot[applied={T|T < 7298541230020136960}]
I20260619 12:00:17.490207 8877 mvcc.cc:205] Tried to move back new op lower bound from 7298541230020136960 to 7298541229910614016. Current Snapshot: MvccSnapshot[applied={T|T < 7298541230020136960 or (T in {7298541230020136960})}]
I20260619 12:00:17.492504 8351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:17.492631 8351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:17.493643 8351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Sending DeleteTablet for 3 replicas of tablet 9c3c1d8993454325ae27bb6db4e47a9c
I20260619 12:00:17.494083 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3c1d8993454325ae27bb6db4e47a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:17 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44064
I20260619 12:00:17.494067 8769 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3c1d8993454325ae27bb6db4e47a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:17 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:53406
I20260619 12:00:17.494086 8488 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3c1d8993454325ae27bb6db4e47a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:17 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44196
I20260619 12:00:17.494376 8886 tablet_replica.cc:330] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: stopping tablet replica
I20260619 12:00:17.494499 8886 raft_consensus.cc:2229] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:17.494664 8886 raft_consensus.cc:2258] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:17.494629 8351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:17.494774 8351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:17.494820 8888 tablet_replica.cc:330] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: stopping tablet replica
I20260619 12:00:17.494905 8888 raft_consensus.cc:2229] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:17.495028 8888 raft_consensus.cc:2258] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:17.495221 8887 tablet_replica.cc:330] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: stopping tablet replica
I20260619 12:00:17.495314 8887 raft_consensus.cc:2229] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:00:17.495405 8887 raft_consensus.cc:2258] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:17.495811 8886 ts_tablet_manager.cc:1905] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:17.495903 8888 ts_tablet_manager.cc:1905] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:17.496483 8488 tablet_service.cc:1467] Processing CreateTablet for tablet 53ba2eb0ef4649c09077703d1d058644 (DEFAULT_TABLE table=test-table [id=1c287e99c8bd4dae87b6bd9556bf0da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:17.496610 8488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ba2eb0ef4649c09077703d1d058644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.496806 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 53ba2eb0ef4649c09077703d1d058644 (DEFAULT_TABLE table=test-table [id=1c287e99c8bd4dae87b6bd9556bf0da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:17.496933 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ba2eb0ef4649c09077703d1d058644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.497928 8886 ts_tablet_manager.cc:1918] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:17.498004 8888 ts_tablet_manager.cc:1918] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:17.498041 8886 log.cc:1196] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/9c3c1d8993454325ae27bb6db4e47a9c
I20260619 12:00:17.498077 8888 log.cc:1196] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/9c3c1d8993454325ae27bb6db4e47a9c
I20260619 12:00:17.498358 8886 ts_tablet_manager.cc:1939] T 9c3c1d8993454325ae27bb6db4e47a9c P 850ce0b5cea94c68bc86936bfefb2008: Deleting consensus metadata
I20260619 12:00:17.498420 8888 ts_tablet_manager.cc:1939] T 9c3c1d8993454325ae27bb6db4e47a9c P a7d0d7d11415485c804e2e3268ab737b: Deleting consensus metadata
I20260619 12:00:17.498720 8863 tablet_bootstrap.cc:492] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: Bootstrap starting.
I20260619 12:00:17.498770 8865 tablet_bootstrap.cc:492] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: Bootstrap starting.
I20260619 12:00:17.499054 8887 ts_tablet_manager.cc:1905] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:17.499153 8865 tablet_bootstrap.cc:654] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.499153 8863 tablet_bootstrap.cc:654] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.499631 8339 catalog_manager.cc:4841] TS 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505): tablet 9c3c1d8993454325ae27bb6db4e47a9c (table test-table [id=f0e317c05ac0435ab87f9bd20b4238a1]) successfully deleted
I20260619 12:00:17.499814 8865 tablet_bootstrap.cc:492] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: No bootstrap required, opened a new log
I20260619 12:00:17.499816 8769 tablet_service.cc:1467] Processing CreateTablet for tablet 53ba2eb0ef4649c09077703d1d058644 (DEFAULT_TABLE table=test-table [id=1c287e99c8bd4dae87b6bd9556bf0da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:17.499866 8865 ts_tablet_manager.cc:1397] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:17.500005 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ba2eb0ef4649c09077703d1d058644. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.500272 8865 raft_consensus.cc:348] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.500371 8865 raft_consensus.cc:374] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.500391 8865 raft_consensus.cc:729] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d0d7d11415485c804e2e3268ab737b, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.500440 8865 consensus_queue.cc:260] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.500550 8865 ts_tablet_manager.cc:1428] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:17.501291 8863 tablet_bootstrap.cc:492] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: No bootstrap required, opened a new log
I20260619 12:00:17.501345 8863 ts_tablet_manager.cc:1397] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 12:00:17.501502 8863 raft_consensus.cc:348] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.501595 8863 raft_consensus.cc:374] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.501616 8863 raft_consensus.cc:729] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850ce0b5cea94c68bc86936bfefb2008, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.501658 8863 consensus_queue.cc:260] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.501802 8863 ts_tablet_manager.cc:1428] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:17.501832 8887 ts_tablet_manager.cc:1918] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:17.501921 8887 log.cc:1196] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/9c3c1d8993454325ae27bb6db4e47a9c
I20260619 12:00:17.502154 8887 ts_tablet_manager.cc:1939] T 9c3c1d8993454325ae27bb6db4e47a9c P 4c13ac479b9b43aa8f8a659d418462a4: Deleting consensus metadata
I20260619 12:00:17.502226 8864 tablet_bootstrap.cc:492] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: Bootstrap starting.
I20260619 12:00:17.502714 8864 tablet_bootstrap.cc:654] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.502714 8337 catalog_manager.cc:4841] TS 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215): tablet 9c3c1d8993454325ae27bb6db4e47a9c (table test-table [id=f0e317c05ac0435ab87f9bd20b4238a1]) successfully deleted
I20260619 12:00:17.503500 8864 tablet_bootstrap.cc:492] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: No bootstrap required, opened a new log
I20260619 12:00:17.503547 8864 ts_tablet_manager.cc:1397] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:17.503711 8864 raft_consensus.cc:348] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.503782 8864 raft_consensus.cc:374] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.503798 8864 raft_consensus.cc:729] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.503859 8864 consensus_queue.cc:260] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.503947 8864 ts_tablet_manager.cc:1428] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:17.504966 8338 catalog_manager.cc:4841] TS a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565): tablet 9c3c1d8993454325ae27bb6db4e47a9c (table test-table [id=f0e317c05ac0435ab87f9bd20b4238a1]) successfully deleted
I20260619 12:00:17.790110 8873 raft_consensus.cc:482] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:17.790251 8873 raft_consensus.cc:504] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.790472 8873 leader_election.cc:288] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565), 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505)
I20260619 12:00:17.790814 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ba2eb0ef4649c09077703d1d058644" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008" is_pre_election: true
I20260619 12:00:17.790961 8649 raft_consensus.cc:2454] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 0.
I20260619 12:00:17.790896 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ba2eb0ef4649c09077703d1d058644" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b" is_pre_election: true
I20260619 12:00:17.791020 8499 raft_consensus.cc:2454] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 0.
I20260619 12:00:17.791221 8723 leader_election.cc:302] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c13ac479b9b43aa8f8a659d418462a4, 850ce0b5cea94c68bc86936bfefb2008; no voters:
I20260619 12:00:17.791359 8873 raft_consensus.cc:2781] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:17.791419 8873 raft_consensus.cc:482] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:17.791436 8873 raft_consensus.cc:3037] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:17.792178 8873 raft_consensus.cc:504] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.792351 8873 leader_election.cc:288] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 election: Requested vote from peers a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565), 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505)
I20260619 12:00:17.792562 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ba2eb0ef4649c09077703d1d058644" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b"
I20260619 12:00:17.792668 8499 raft_consensus.cc:3037] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:17.792935 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ba2eb0ef4649c09077703d1d058644" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008"
I20260619 12:00:17.793009 8649 raft_consensus.cc:3037] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:17.793551 8499 raft_consensus.cc:2454] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 1.
I20260619 12:00:17.793676 8649 raft_consensus.cc:2454] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 1.
I20260619 12:00:17.793903 8723 leader_election.cc:302] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c13ac479b9b43aa8f8a659d418462a4, 850ce0b5cea94c68bc86936bfefb2008; no voters:
I20260619 12:00:17.794056 8873 raft_consensus.cc:2781] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:17.794133 8873 raft_consensus.cc:686] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Running, Role: LEADER
I20260619 12:00:17.794198 8873 consensus_queue.cc:237] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.794984 8352 catalog_manager.cc:5495] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195). New cstate: current_term: 1 leader_uuid: "4c13ac479b9b43aa8f8a659d418462a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } health_report { overall_health: HEALTHY } } }
Jun 19 12:00:17 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870416, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.254@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:17.878345 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.869555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39449 (local address 127.0.0.1:51872)
0619 12:00:17.869796 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:17.869808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:17.869874 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:17.870151 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:17.870152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:17.870510 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:17.870574 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.870576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.871379 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.871380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.872078 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:17.872080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.872516 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.872517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:17.872568 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.873277 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:17.873281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:17.874810 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:17.876766 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:17.876776 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:17.876779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:17.877144 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:17.877409 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:17.877409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:17.877410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:17.877474 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:17.877946 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:17.877956 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:17.878022 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 12:00:17.878100 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:00:17.878241 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260619 12:00:17.878454 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:17.869658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.0.203.254:39449)
0619 12:00:17.869817 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:17.869819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:17.869861 (+ 42us) server_negotiation.cc:407] Connection header received
0619 12:00:17.869968 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:17.869970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:17.869981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:17.870095 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:17.870689 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.871253 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.872185 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:17.872412 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:17.872477 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:17.874931 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:17.874933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:17.874933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:17.874942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:17.876601 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.877259 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.877260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.877261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.877331 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:17.877569 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:17.877570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:17.877570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:17.877710 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:17.877779 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:17.878143 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 12:00:17.878346 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260619 12:00:17.881341 8352 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:51872:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 12:00:17.881669 8352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:17.883780 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc32ccc2df34fc2a81f8decd9fb26c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5e5f1fcf17b4f29bc26e789d2f77373]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:00:17.883780 8769 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc32ccc2df34fc2a81f8decd9fb26c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5e5f1fcf17b4f29bc26e789d2f77373]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:00:17.883832 8488 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc32ccc2df34fc2a81f8decd9fb26c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5e5f1fcf17b4f29bc26e789d2f77373]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:00:17.883955 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc32ccc2df34fc2a81f8decd9fb26c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.883955 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc32ccc2df34fc2a81f8decd9fb26c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.884059 8488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc32ccc2df34fc2a81f8decd9fb26c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:17.885699 8863 tablet_bootstrap.cc:492] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: Bootstrap starting.
I20260619 12:00:17.885812 8864 tablet_bootstrap.cc:492] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: Bootstrap starting.
I20260619 12:00:17.886199 8865 tablet_bootstrap.cc:492] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: Bootstrap starting.
I20260619 12:00:17.886217 8864 tablet_bootstrap.cc:654] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.886255 8863 tablet_bootstrap.cc:654] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.886693 8865 tablet_bootstrap.cc:654] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:17.887147 8863 tablet_bootstrap.cc:492] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: No bootstrap required, opened a new log
I20260619 12:00:17.887204 8863 ts_tablet_manager.cc:1397] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:00:17.887405 8863 raft_consensus.cc:348] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.887485 8865 tablet_bootstrap.cc:492] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: No bootstrap required, opened a new log
I20260619 12:00:17.887533 8863 raft_consensus.cc:374] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.887553 8865 ts_tablet_manager.cc:1397] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:17.887565 8863 raft_consensus.cc:729] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850ce0b5cea94c68bc86936bfefb2008, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.887631 8863 consensus_queue.cc:260] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.887758 8863 ts_tablet_manager.cc:1428] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:17.887742 8865 raft_consensus.cc:348] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.887833 8865 raft_consensus.cc:374] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.887866 8865 raft_consensus.cc:729] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d0d7d11415485c804e2e3268ab737b, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.887924 8865 consensus_queue.cc:260] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.887960 8870 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:17.888027 8865 ts_tablet_manager.cc:1428] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 12:00:17.888051 8870 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:00:17.888291 8864 tablet_bootstrap.cc:492] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: No bootstrap required, opened a new log
I20260619 12:00:17.888346 8864 ts_tablet_manager.cc:1397] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 12:00:17.888459 8864 raft_consensus.cc:348] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.888525 8864 raft_consensus.cc:374] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:17.888545 8864 raft_consensus.cc:729] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Initialized, Role: FOLLOWER
I20260619 12:00:17.888584 8864 consensus_queue.cc:260] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:17.888696 8864 ts_tablet_manager.cc:1428] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:17.888859 8871 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:17.888960 8871 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:00:17.889079 8873 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:17.889159 8873 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:00:18.041723 8870 raft_consensus.cc:482] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:18.041877 8870 raft_consensus.cc:504] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.042301 8870 leader_election.cc:288] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565), 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215)
I20260619 12:00:18.045811 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.042607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:45215 (local address 127.0.203.194:51703)
0619 12:00:18.042733 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.042746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:18.042838 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:18.043122 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:18.043123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:18.043128 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:18.043240 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.043243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.044032 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.044034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.044941 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.044943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.045542 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.045544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.045602 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.045607 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:18.045661 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 12:00:18.045697 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260619 12:00:18.045814 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.042663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:51703 (local address 127.0.203.195:45215)
0619 12:00:18.042798 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.042805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.042815 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:18.042952 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.042953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.042965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.043053 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:18.043329 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.043896 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.045070 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.045452 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.045511 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.045519 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.045686 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.045739 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260619 12:00:18.045900 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.042569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:51275 (local address 127.0.203.193:46565)
0619 12:00:18.042742 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.042744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.042781 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:00:18.042820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.042820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.042834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.042925 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:18.043159 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.043805 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.045051 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.045474 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.045577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.045587 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.045758 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.045805 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260619 12:00:18.045966 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.042496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:46565 (local address 127.0.203.194:51275)
0619 12:00:18.042638 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.042651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:18.042716 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:18.042952 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:18.042953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:18.042957 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:18.043026 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.043029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.043896 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.043898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.044929 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.044932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.045596 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.045597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.045667 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.045672 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:18.045728 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 12:00:18.045804 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260619 12:00:18.046376 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc32ccc2df34fc2a81f8decd9fb26c2" candidate_uuid: "850ce0b5cea94c68bc86936bfefb2008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c13ac479b9b43aa8f8a659d418462a4" is_pre_election: true
I20260619 12:00:18.046434 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc32ccc2df34fc2a81f8decd9fb26c2" candidate_uuid: "850ce0b5cea94c68bc86936bfefb2008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b" is_pre_election: true
I20260619 12:00:18.046518 8499 raft_consensus.cc:2454] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 850ce0b5cea94c68bc86936bfefb2008 in term 0.
I20260619 12:00:18.046540 8789 raft_consensus.cc:2454] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 850ce0b5cea94c68bc86936bfefb2008 in term 0.
I20260619 12:00:18.046705 8582 leader_election.cc:302] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 850ce0b5cea94c68bc86936bfefb2008, a7d0d7d11415485c804e2e3268ab737b; no voters:
I20260619 12:00:18.046850 8870 raft_consensus.cc:2781] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:18.046896 8870 raft_consensus.cc:482] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:18.046911 8870 raft_consensus.cc:3037] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.047617 8870 raft_consensus.cc:504] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.047772 8870 leader_election.cc:288] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [CANDIDATE]: Term 1 election: Requested vote from peers a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565), 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215)
I20260619 12:00:18.047901 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc32ccc2df34fc2a81f8decd9fb26c2" candidate_uuid: "850ce0b5cea94c68bc86936bfefb2008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b"
I20260619 12:00:18.047960 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc32ccc2df34fc2a81f8decd9fb26c2" candidate_uuid: "850ce0b5cea94c68bc86936bfefb2008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c13ac479b9b43aa8f8a659d418462a4"
I20260619 12:00:18.048002 8499 raft_consensus.cc:3037] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.048015 8789 raft_consensus.cc:3037] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.048946 8499 raft_consensus.cc:2454] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 850ce0b5cea94c68bc86936bfefb2008 in term 1.
I20260619 12:00:18.049149 8789 raft_consensus.cc:2454] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 850ce0b5cea94c68bc86936bfefb2008 in term 1.
I20260619 12:00:18.049247 8582 leader_election.cc:302] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 850ce0b5cea94c68bc86936bfefb2008, a7d0d7d11415485c804e2e3268ab737b; no voters:
I20260619 12:00:18.049458 8870 raft_consensus.cc:2781] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:18.049631 8870 raft_consensus.cc:686] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 LEADER]: Becoming Leader. State: Replica: 850ce0b5cea94c68bc86936bfefb2008, State: Running, Role: LEADER
I20260619 12:00:18.049734 8870 consensus_queue.cc:237] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.050143 8907 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: TxnStatusTablet state changed. Reason: New leader 850ce0b5cea94c68bc86936bfefb2008. Latest consensus state: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:18.050231 8907 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:00:18.050520 8909 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 12:00:18.050714 8909 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 12:00:18.050652 8351 catalog_manager.cc:5495] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194). New cstate: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20260619 12:00:18.075630 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.072392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.0.203.194:37505)
0619 12:00:18.072618 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.072620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.072627 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:00:18.072695 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.072696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.072717 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.072813 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:18.072995 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.073618 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.074400 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.074620 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.074709 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.075048 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.075157 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.075224 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.075518 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.075540 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
I20260619 12:00:18.075700 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.072300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37505 (local address 127.0.0.1:44072)
0619 12:00:18.072387 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.072400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:18.072452 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:18.072839 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:18.072840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:18.072841 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:00:18.072896 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.072899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.073711 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.073712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.074307 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.074309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.074736 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.074738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.074855 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.074894 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.075241 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.075250 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:00:18.075510 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 12:00:18.075595 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 12:00:18.077239 8499 raft_consensus.cc:1261] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Refusing update from remote peer 850ce0b5cea94c68bc86936bfefb2008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.077412 8789 raft_consensus.cc:1261] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Refusing update from remote peer 850ce0b5cea94c68bc86936bfefb2008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.077672 8907 consensus_queue.cc:1035] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.077838 8870 consensus_queue.cc:1035] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.078621 8873 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: TxnStatusTablet state changed. Reason: New leader 850ce0b5cea94c68bc86936bfefb2008. Latest consensus state: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:18.078621 8871 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: TxnStatusTablet state changed. Reason: New leader 850ce0b5cea94c68bc86936bfefb2008. Latest consensus state: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:18.078727 8873 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:00:18.078727 8871 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:00:18.080034 8870 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:18.080132 8870 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:00:18.080184 8870 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:18.080224 8870 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 850ce0b5cea94c68bc86936bfefb2008: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:00:18.080332 8873 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:18.080430 8873 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P 4c13ac479b9b43aa8f8a659d418462a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:00:18.080444 8913 mvcc.cc:205] Tried to move back new op lower bound from 7298541232441921536 to 7298541232332558336. Current Snapshot: MvccSnapshot[applied={T|T < 7298541232441921536 or (T in {7298541232441921536})}]
I20260619 12:00:18.081604 8871 tablet_replica.cc:439] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "850ce0b5cea94c68bc86936bfefb2008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } } }
I20260619 12:00:18.081719 8871 tablet_replica.cc:442] T 6bc32ccc2df34fc2a81f8decd9fb26c2 P a7d0d7d11415485c804e2e3268ab737b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:00:18.087308 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.084449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:52325 (local address 127.0.203.194:37505)
0619 12:00:18.084496 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.084498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.084588 (+ 90us) server_negotiation.cc:407] Connection header received
0619 12:00:18.084667 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.084668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.084682 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.084740 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:18.084897 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.085709 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.086509 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.086819 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.086895 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.087050 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.087091 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.087147 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.087208 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.087231 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:00:18.087329 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.084353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37505 (local address 127.0.203.195:52325)
0619 12:00:18.084505 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.084516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:18.084580 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:18.084767 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:18.084767 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:18.084769 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:00:18.084820 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.084822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.085807 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.085808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.086423 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.086425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.086904 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.086905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.086949 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.086962 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.087158 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.087160 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 12:00:18.087201 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 12:00:18.087244 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260619 12:00:18.089910 8649 raft_consensus.cc:1261] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Refusing update from remote peer 4c13ac479b9b43aa8f8a659d418462a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.090045 8499 raft_consensus.cc:1261] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Refusing update from remote peer 4c13ac479b9b43aa8f8a659d418462a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.090160 8873 consensus_queue.cc:1035] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.090263 8869 consensus_queue.cc:1035] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.099570 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.096920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:33523 (local address 127.0.203.195:45215)
0619 12:00:18.096966 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.096968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.097161 (+ 193us) server_negotiation.cc:407] Connection header received
0619 12:00:18.097202 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.097203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.097215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.097278 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:18.097456 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.097947 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.098755 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.098952 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.099032 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.099316 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.099344 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.099402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.099463 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.099487 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:00:18.099844 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.096850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:45215 (local address 127.0.203.194:33523)
0619 12:00:18.097071 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.097084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:18.097152 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:18.097287 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:18.097288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:18.097291 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:00:18.097357 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.097359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.098045 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.098046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.098667 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.098669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.099041 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.099042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.099104 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.099127 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.099416 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.099418 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 12:00:18.099461 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 12:00:18.099513 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":116,"threads_started":1}
I20260619 12:00:18.107390 8351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:18.107506 8351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:18.108273 8351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Sending DeleteTablet for 3 replicas of tablet 53ba2eb0ef4649c09077703d1d058644
I20260619 12:00:18.108578 8769 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ba2eb0ef4649c09077703d1d058644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:53406
I20260619 12:00:18.108752 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ba2eb0ef4649c09077703d1d058644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44064
I20260619 12:00:18.108978 8931 tablet_replica.cc:330] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: stopping tablet replica
I20260619 12:00:18.109041 8931 raft_consensus.cc:2229] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:00:18.109076 8932 tablet_replica.cc:330] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: stopping tablet replica
I20260619 12:00:18.109110 8931 raft_consensus.cc:2258] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.109144 8932 raft_consensus.cc:2229] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:18.109187 8932 raft_consensus.cc:2258] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.109301 8488 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ba2eb0ef4649c09077703d1d058644 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44196
I20260619 12:00:18.109414 8931 ts_tablet_manager.cc:1905] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.109519 8932 ts_tablet_manager.cc:1905] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.110869 8933 tablet_replica.cc:330] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: stopping tablet replica
I20260619 12:00:18.111002 8933 raft_consensus.cc:2229] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:18.111063 8933 raft_consensus.cc:2258] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.111142 8931 ts_tablet_manager.cc:1918] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:00:18.111232 8931 log.cc:1196] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/53ba2eb0ef4649c09077703d1d058644
I20260619 12:00:18.111364 8933 ts_tablet_manager.cc:1905] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.111624 8931 ts_tablet_manager.cc:1939] T 53ba2eb0ef4649c09077703d1d058644 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting consensus metadata
I20260619 12:00:18.112038 8337 catalog_manager.cc:4841] TS 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215): tablet 53ba2eb0ef4649c09077703d1d058644 (table test-table [id=1c287e99c8bd4dae87b6bd9556bf0da2]) successfully deleted
I20260619 12:00:18.113112 8933 ts_tablet_manager.cc:1918] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:00:18.113186 8933 log.cc:1196] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/53ba2eb0ef4649c09077703d1d058644
I20260619 12:00:18.113425 8933 ts_tablet_manager.cc:1939] T 53ba2eb0ef4649c09077703d1d058644 P a7d0d7d11415485c804e2e3268ab737b: Deleting consensus metadata
I20260619 12:00:18.114085 8338 catalog_manager.cc:4841] TS a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565): tablet 53ba2eb0ef4649c09077703d1d058644 (table test-table [id=1c287e99c8bd4dae87b6bd9556bf0da2]) successfully deleted
I20260619 12:00:18.115150 8932 ts_tablet_manager.cc:1918] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:00:18.115229 8932 log.cc:1196] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/53ba2eb0ef4649c09077703d1d058644
I20260619 12:00:18.115489 8932 ts_tablet_manager.cc:1939] T 53ba2eb0ef4649c09077703d1d058644 P 850ce0b5cea94c68bc86936bfefb2008: Deleting consensus metadata
I20260619 12:00:18.115876 8339 catalog_manager.cc:4841] TS 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505): tablet 53ba2eb0ef4649c09077703d1d058644 (table test-table [id=1c287e99c8bd4dae87b6bd9556bf0da2]) successfully deleted
Jun 19 12:00:18 dist-test-slave-fbq8 krb5kdc[8307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:00:18 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:18 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:00:18.142709 8899 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:18.136983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.203.254:39449)
0619 12:00:18.137093 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.137095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.137180 (+ 85us) server_negotiation.cc:407] Connection header received
0619 12:00:18.137289 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.137290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.137304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.137370 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:18.137968 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.138569 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.139380 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.139580 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.139660 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.142613 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51878: BlockingRecv error: recv got EOF from 127.0.0.1:51878 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 12:00:18 dist-test-slave-fbq8 krb5kdc[8307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM
I20260619 12:00:18.151644 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.143781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.0.203.254:39449)
0619 12:00:18.143816 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.143819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.144023 (+ 204us) server_negotiation.cc:407] Connection header received
0619 12:00:18.144085 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.144086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.144099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.144154 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:18.144714 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.145187 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.145875 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.146070 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.146143 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.148811 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:18.148813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:18.148813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:18.148820 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:18.150363 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:18.150871 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:18.150872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:18.150873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:18.150925 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:18.151158 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:18.151159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:18.151159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:18.151251 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:18.151340 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.151514 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.151578 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 12:00:18.153577 8351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:18.153769 8351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:18.155323 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 1e13dc632b164cb197173b81cfe2f4f9 (DEFAULT_TABLE table=test-table [id=bf97e50a616545b5a4ab4f5e9d921886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:18.155308 8488 tablet_service.cc:1467] Processing CreateTablet for tablet 1e13dc632b164cb197173b81cfe2f4f9 (DEFAULT_TABLE table=test-table [id=bf97e50a616545b5a4ab4f5e9d921886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:18.155385 8769 tablet_service.cc:1467] Processing CreateTablet for tablet 1e13dc632b164cb197173b81cfe2f4f9 (DEFAULT_TABLE table=test-table [id=bf97e50a616545b5a4ab4f5e9d921886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:18.155478 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e13dc632b164cb197173b81cfe2f4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:18.155491 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e13dc632b164cb197173b81cfe2f4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:18.155493 8488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e13dc632b164cb197173b81cfe2f4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:18.157033 8864 tablet_bootstrap.cc:492] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: Bootstrap starting.
I20260619 12:00:18.157419 8864 tablet_bootstrap.cc:654] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:18.157512 8863 tablet_bootstrap.cc:492] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: Bootstrap starting.
I20260619 12:00:18.157698 8865 tablet_bootstrap.cc:492] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: Bootstrap starting.
I20260619 12:00:18.158004 8863 tablet_bootstrap.cc:654] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:18.158155 8865 tablet_bootstrap.cc:654] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:18.158751 8863 tablet_bootstrap.cc:492] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: No bootstrap required, opened a new log
I20260619 12:00:18.158805 8863 ts_tablet_manager.cc:1397] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:18.158985 8863 raft_consensus.cc:348] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.159086 8863 raft_consensus.cc:374] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:18.159111 8863 raft_consensus.cc:729] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850ce0b5cea94c68bc86936bfefb2008, State: Initialized, Role: FOLLOWER
I20260619 12:00:18.159132 8864 tablet_bootstrap.cc:492] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: No bootstrap required, opened a new log
I20260619 12:00:18.159175 8864 ts_tablet_manager.cc:1397] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:00:18.159171 8863 consensus_queue.cc:260] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.159268 8863 ts_tablet_manager.cc:1428] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:18.159293 8864 raft_consensus.cc:348] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.159374 8864 raft_consensus.cc:374] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:18.159401 8864 raft_consensus.cc:729] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Initialized, Role: FOLLOWER
I20260619 12:00:18.159453 8864 consensus_queue.cc:260] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.159543 8864 ts_tablet_manager.cc:1428] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:18.159991 8865 tablet_bootstrap.cc:492] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: No bootstrap required, opened a new log
I20260619 12:00:18.160058 8865 ts_tablet_manager.cc:1397] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 12:00:18.160228 8865 raft_consensus.cc:348] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.160310 8865 raft_consensus.cc:374] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:18.160332 8865 raft_consensus.cc:729] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d0d7d11415485c804e2e3268ab737b, State: Initialized, Role: FOLLOWER
I20260619 12:00:18.160390 8865 consensus_queue.cc:260] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.160504 8865 ts_tablet_manager.cc:1428] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 12:00:18.181779 8871 raft_consensus.cc:482] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:18.181905 8871 raft_consensus.cc:504] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.182303 8871 leader_election.cc:288] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505), 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215)
I20260619 12:00:18.186551 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.182800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:44615 (local address 127.0.203.195:45215)
0619 12:00:18.182861 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.182864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.182874 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:18.182911 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.182912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.182926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.182993 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:18.183306 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.184038 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.185467 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.185964 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.186044 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.186055 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.186417 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.186464 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:00:18.186635 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.182535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:45215 (local address 127.0.203.193:44615)
0619 12:00:18.182731 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.182744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:18.182816 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:18.183055 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:18.183057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:18.183060 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:18.183172 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.183175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.184361 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.184362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.185361 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.185363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.186095 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.186097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.186220 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.186225 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:00:18.186414 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 12:00:18.186522 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":133,"threads_started":1}
I20260619 12:00:18.186762 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.182554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:51703 (local address 127.0.203.194:37505)
0619 12:00:18.182607 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.182609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.182807 (+ 198us) server_negotiation.cc:407] Connection header received
0619 12:00:18.182912 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.182913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.182926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.182955 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:18.183199 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.183997 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.185852 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.186318 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.186414 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.186422 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.186637 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.186683 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 12:00:18.186959 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e13dc632b164cb197173b81cfe2f4f9" candidate_uuid: "a7d0d7d11415485c804e2e3268ab737b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c13ac479b9b43aa8f8a659d418462a4" is_pre_election: true
I20260619 12:00:18.187088 8789 raft_consensus.cc:2454] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7d0d7d11415485c804e2e3268ab737b in term 0.
I20260619 12:00:18.187315 8441 leader_election.cc:302] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c13ac479b9b43aa8f8a659d418462a4, a7d0d7d11415485c804e2e3268ab737b; no voters:
I20260619 12:00:18.187480 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.182496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37505 (local address 127.0.203.193:51703)
0619 12:00:18.182728 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.182740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:18.182832 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:18.183030 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:18.183031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:18.183032 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:18.183111 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.183112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.184149 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.184150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.185578 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:18.185581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.186459 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.186461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:18.186551 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.186555 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:18.187279 (+ 724us) client_negotiation.cc:240] Negotiation successful
0619 12:00:18.187337 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":44,"thread_start_us":125,"threads_started":1}
I20260619 12:00:18.187484 8871 raft_consensus.cc:2781] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:18.187608 8871 raft_consensus.cc:482] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:18.187638 8871 raft_consensus.cc:3037] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.187805 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e13dc632b164cb197173b81cfe2f4f9" candidate_uuid: "a7d0d7d11415485c804e2e3268ab737b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008" is_pre_election: true
I20260619 12:00:18.187912 8649 raft_consensus.cc:2454] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7d0d7d11415485c804e2e3268ab737b in term 0.
I20260619 12:00:18.188606 8871 raft_consensus.cc:504] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.188820 8871 leader_election.cc:288] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [CANDIDATE]: Term 1 election: Requested vote from peers 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505), 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215)
I20260619 12:00:18.188956 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e13dc632b164cb197173b81cfe2f4f9" candidate_uuid: "a7d0d7d11415485c804e2e3268ab737b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008"
I20260619 12:00:18.189028 8649 raft_consensus.cc:3037] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.189080 8789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e13dc632b164cb197173b81cfe2f4f9" candidate_uuid: "a7d0d7d11415485c804e2e3268ab737b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c13ac479b9b43aa8f8a659d418462a4"
I20260619 12:00:18.189164 8789 raft_consensus.cc:3037] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.189690 8649 raft_consensus.cc:2454] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7d0d7d11415485c804e2e3268ab737b in term 1.
I20260619 12:00:18.189898 8443 leader_election.cc:302] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 850ce0b5cea94c68bc86936bfefb2008, a7d0d7d11415485c804e2e3268ab737b; no voters:
I20260619 12:00:18.189988 8871 raft_consensus.cc:2781] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:18.190166 8871 raft_consensus.cc:686] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 1 LEADER]: Becoming Leader. State: Replica: a7d0d7d11415485c804e2e3268ab737b, State: Running, Role: LEADER
I20260619 12:00:18.190202 8789 raft_consensus.cc:2454] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7d0d7d11415485c804e2e3268ab737b in term 1.
I20260619 12:00:18.190241 8871 consensus_queue.cc:237] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.191032 8350 catalog_manager.cc:5495] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b reported cstate change: term changed from 0 to 1, leader changed from <none> to a7d0d7d11415485c804e2e3268ab737b (127.0.203.193). New cstate: current_term: 1 leader_uuid: "a7d0d7d11415485c804e2e3268ab737b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } health_report { overall_health: UNKNOWN } } }
I20260619 12:00:18.200037 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.197205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.0.203.193:46565)
0619 12:00:18.197277 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.197280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.197291 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:18.197399 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.197400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.197423 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.197528 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:18.197695 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.198237 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.199104 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.199335 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.199421 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.199636 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.199765 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.199863 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.199910 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.199936 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 12:00:18.201531 8649 raft_consensus.cc:1261] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Refusing update from remote peer a7d0d7d11415485c804e2e3268ab737b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.201516 8789 raft_consensus.cc:1261] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Refusing update from remote peer a7d0d7d11415485c804e2e3268ab737b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.201838 8951 consensus_queue.cc:1035] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.201970 8871 consensus_queue.cc:1035] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [LEADER]: Connected to new peer: Peer: permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.206079 8351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:18.206219 8351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:18.206964 8351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Sending DeleteTablet for 3 replicas of tablet 1e13dc632b164cb197173b81cfe2f4f9
I20260619 12:00:18.207252 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e13dc632b164cb197173b81cfe2f4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44064
I20260619 12:00:18.207361 8932 tablet_replica.cc:330] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: stopping tablet replica
I20260619 12:00:18.207330 8488 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e13dc632b164cb197173b81cfe2f4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44196
I20260619 12:00:18.207413 8932 raft_consensus.cc:2229] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:18.207455 8933 tablet_replica.cc:330] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: stopping tablet replica
I20260619 12:00:18.207471 8932 raft_consensus.cc:2258] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.207513 8933 raft_consensus.cc:2229] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:00:18.207631 8933 raft_consensus.cc:2258] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.207619 8769 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e13dc632b164cb197173b81cfe2f4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:53406
I20260619 12:00:18.207702 8931 tablet_replica.cc:330] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: stopping tablet replica
I20260619 12:00:18.207752 8931 raft_consensus.cc:2229] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:18.207800 8931 raft_consensus.cc:2258] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.207981 8933 ts_tablet_manager.cc:1905] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.208374 8351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:18.208532 8351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:18.209473 8931 ts_tablet_manager.cc:1905] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.209743 8933 ts_tablet_manager.cc:1918] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:18.209789 8933 log.cc:1196] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/1e13dc632b164cb197173b81cfe2f4f9
I20260619 12:00:18.209990 8933 ts_tablet_manager.cc:1939] T 1e13dc632b164cb197173b81cfe2f4f9 P a7d0d7d11415485c804e2e3268ab737b: Deleting consensus metadata
I20260619 12:00:18.210479 8338 catalog_manager.cc:4841] TS a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565): tablet 1e13dc632b164cb197173b81cfe2f4f9 (table test-table [id=bf97e50a616545b5a4ab4f5e9d921886]) successfully deleted
I20260619 12:00:18.211050 8931 ts_tablet_manager.cc:1918] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:18.211098 8931 log.cc:1196] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/1e13dc632b164cb197173b81cfe2f4f9
I20260619 12:00:18.211302 8931 ts_tablet_manager.cc:1939] T 1e13dc632b164cb197173b81cfe2f4f9 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting consensus metadata
I20260619 12:00:18.211922 8932 ts_tablet_manager.cc:1905] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.212268 8488 tablet_service.cc:1467] Processing CreateTablet for tablet ef6b704e1e034a52b20b16a6a52139c6 (DEFAULT_TABLE table=test-table [id=d2bf2453c2dc4b9fb1d6da2accad226e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:18.212416 8488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6b704e1e034a52b20b16a6a52139c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:18.213389 8932 ts_tablet_manager.cc:1918] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:18.213442 8932 log.cc:1196] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/1e13dc632b164cb197173b81cfe2f4f9
I20260619 12:00:18.213654 8932 ts_tablet_manager.cc:1939] T 1e13dc632b164cb197173b81cfe2f4f9 P 850ce0b5cea94c68bc86936bfefb2008: Deleting consensus metadata
I20260619 12:00:18.213845 8865 tablet_bootstrap.cc:492] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b: Bootstrap starting.
I20260619 12:00:18.214107 8769 tablet_service.cc:1467] Processing CreateTablet for tablet ef6b704e1e034a52b20b16a6a52139c6 (DEFAULT_TABLE table=test-table [id=d2bf2453c2dc4b9fb1d6da2accad226e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:18.214234 8769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6b704e1e034a52b20b16a6a52139c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:18.214344 8865 tablet_bootstrap.cc:654] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:18.215498 8864 tablet_bootstrap.cc:492] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: Bootstrap starting.
I20260619 12:00:18.215796 8864 tablet_bootstrap.cc:654] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:18.216077 8629 tablet_service.cc:1467] Processing CreateTablet for tablet ef6b704e1e034a52b20b16a6a52139c6 (DEFAULT_TABLE table=test-table [id=d2bf2453c2dc4b9fb1d6da2accad226e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:18.216176 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6b704e1e034a52b20b16a6a52139c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:18.216429 8864 tablet_bootstrap.cc:492] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: No bootstrap required, opened a new log
I20260619 12:00:18.216480 8864 ts_tablet_manager.cc:1397] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:18.216611 8864 raft_consensus.cc:348] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.216715 8864 raft_consensus.cc:374] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:18.216737 8864 raft_consensus.cc:729] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Initialized, Role: FOLLOWER
I20260619 12:00:18.216789 8864 consensus_queue.cc:260] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.216876 8864 ts_tablet_manager.cc:1428] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 12:00:18.217419 8863 tablet_bootstrap.cc:492] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: Bootstrap starting.
I20260619 12:00:18.217689 8339 catalog_manager.cc:4841] TS 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505): tablet 1e13dc632b164cb197173b81cfe2f4f9 (table test-table [id=bf97e50a616545b5a4ab4f5e9d921886]) successfully deleted
I20260619 12:00:18.218117 8863 tablet_bootstrap.cc:654] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:18.218773 8337 catalog_manager.cc:4841] TS 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215): tablet 1e13dc632b164cb197173b81cfe2f4f9 (table test-table [id=bf97e50a616545b5a4ab4f5e9d921886]) successfully deleted
I20260619 12:00:18.219235 8865 tablet_bootstrap.cc:492] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b: No bootstrap required, opened a new log
I20260619 12:00:18.219292 8865 ts_tablet_manager.cc:1397] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 12:00:18.219468 8865 raft_consensus.cc:348] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.219563 8865 raft_consensus.cc:374] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:18.219583 8865 raft_consensus.cc:729] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d0d7d11415485c804e2e3268ab737b, State: Initialized, Role: FOLLOWER
I20260619 12:00:18.219627 8865 consensus_queue.cc:260] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.219719 8865 ts_tablet_manager.cc:1428] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 12:00:18.221088 8863 tablet_bootstrap.cc:492] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: No bootstrap required, opened a new log
I20260619 12:00:18.221148 8863 ts_tablet_manager.cc:1397] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 12:00:18.221275 8863 raft_consensus.cc:348] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.221342 8863 raft_consensus.cc:374] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:18.221362 8863 raft_consensus.cc:729] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850ce0b5cea94c68bc86936bfefb2008, State: Initialized, Role: FOLLOWER
I20260619 12:00:18.221405 8863 consensus_queue.cc:260] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.221505 8863 ts_tablet_manager.cc:1428] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:18.250247 8873 raft_consensus.cc:482] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:18.250385 8873 raft_consensus.cc:504] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.250602 8873 leader_election.cc:288] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565), 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505)
I20260619 12:00:18.250859 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6b704e1e034a52b20b16a6a52139c6" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008" is_pre_election: true
I20260619 12:00:18.250859 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6b704e1e034a52b20b16a6a52139c6" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b" is_pre_election: true
I20260619 12:00:18.250970 8499 raft_consensus.cc:2454] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 0.
I20260619 12:00:18.250970 8649 raft_consensus.cc:2454] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 0.
I20260619 12:00:18.251155 8723 leader_election.cc:302] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c13ac479b9b43aa8f8a659d418462a4, 850ce0b5cea94c68bc86936bfefb2008; no voters:
I20260619 12:00:18.251294 8873 raft_consensus.cc:2781] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:18.251345 8873 raft_consensus.cc:482] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:18.251362 8873 raft_consensus.cc:3037] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.252211 8873 raft_consensus.cc:504] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.252350 8873 leader_election.cc:288] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 election: Requested vote from peers a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565), 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505)
I20260619 12:00:18.252506 8499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6b704e1e034a52b20b16a6a52139c6" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d0d7d11415485c804e2e3268ab737b"
I20260619 12:00:18.252584 8499 raft_consensus.cc:3037] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.252586 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6b704e1e034a52b20b16a6a52139c6" candidate_uuid: "4c13ac479b9b43aa8f8a659d418462a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850ce0b5cea94c68bc86936bfefb2008"
I20260619 12:00:18.252755 8649 raft_consensus.cc:3037] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:18.253311 8499 raft_consensus.cc:2454] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 1.
I20260619 12:00:18.253502 8722 leader_election.cc:302] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c13ac479b9b43aa8f8a659d418462a4, a7d0d7d11415485c804e2e3268ab737b; no voters:
I20260619 12:00:18.253582 8873 raft_consensus.cc:2781] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:18.253630 8873 raft_consensus.cc:686] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4c13ac479b9b43aa8f8a659d418462a4, State: Running, Role: LEADER
I20260619 12:00:18.253646 8649 raft_consensus.cc:2454] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c13ac479b9b43aa8f8a659d418462a4 in term 1.
I20260619 12:00:18.253692 8873 consensus_queue.cc:237] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } }
I20260619 12:00:18.254487 8351 catalog_manager.cc:5495] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195). New cstate: current_term: 1 leader_uuid: "4c13ac479b9b43aa8f8a659d418462a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c13ac479b9b43aa8f8a659d418462a4" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 45215 } health_report { overall_health: HEALTHY } } }
I20260619 12:00:18.266253 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.263540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.0.203.195:45215)
0619 12:00:18.263604 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:18.263606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:18.263614 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:18.263690 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:18.263691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:18.263706 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:18.263835 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:18.263949 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.264451 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.265475 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:18.265709 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:18.265800 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:18.265961 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:18.266005 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:18.266070 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:18.266132 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 12:00:18.266161 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:00:18.269191 8649 raft_consensus.cc:1261] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Refusing update from remote peer 4c13ac479b9b43aa8f8a659d418462a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.269224 8499 raft_consensus.cc:1261] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Refusing update from remote peer 4c13ac479b9b43aa8f8a659d418462a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:18.269426 8873 consensus_queue.cc:1035] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "850ce0b5cea94c68bc86936bfefb2008" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.269562 8873 consensus_queue.cc:1035] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d0d7d11415485c804e2e3268ab737b" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:18.282449 8351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:18.282565 8351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:18.283560 8351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18b68b05110476eaea7509f70dd29a6: Sending DeleteTablet for 3 replicas of tablet ef6b704e1e034a52b20b16a6a52139c6
I20260619 12:00:18.284237 8769 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6b704e1e034a52b20b16a6a52139c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:53406
I20260619 12:00:18.284242 8488 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6b704e1e034a52b20b16a6a52139c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44196
I20260619 12:00:18.284399 8931 tablet_replica.cc:330] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: stopping tablet replica
I20260619 12:00:18.284475 8931 raft_consensus.cc:2229] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:00:18.284550 8931 raft_consensus.cc:2258] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.284668 8933 tablet_replica.cc:330] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b: stopping tablet replica
I20260619 12:00:18.284726 8933 raft_consensus.cc:2229] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:18.284780 8933 raft_consensus.cc:2258] T ef6b704e1e034a52b20b16a6a52139c6 P a7d0d7d11415485c804e2e3268ab737b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.284785 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6b704e1e034a52b20b16a6a52139c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:18 UTC) from {username='oryx', principal='oryx/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44064
I20260619 12:00:18.284895 8932 tablet_replica.cc:330] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: stopping tablet replica
I20260619 12:00:18.284960 8932 raft_consensus.cc:2229] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:18.285001 8932 raft_consensus.cc:2258] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:18.285503 8932 ts_tablet_manager.cc:1905] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.285507 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 8424
I20260619 12:00:18.287720 8932 ts_tablet_manager.cc:1918] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:00:18.287793 8932 log.cc:1196] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/ef6b704e1e034a52b20b16a6a52139c6
I20260619 12:00:18.287897 8931 ts_tablet_manager.cc:1905] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:18.288029 8932 ts_tablet_manager.cc:1939] T ef6b704e1e034a52b20b16a6a52139c6 P 850ce0b5cea94c68bc86936bfefb2008: Deleting consensus metadata
I20260619 12:00:18.288465 8339 catalog_manager.cc:4841] TS 850ce0b5cea94c68bc86936bfefb2008 (127.0.203.194:37505): tablet ef6b704e1e034a52b20b16a6a52139c6 (table test-table [id=d2bf2453c2dc4b9fb1d6da2accad226e]) successfully deleted
I20260619 12:00:18.290045 8931 ts_tablet_manager.cc:1918] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:00:18.290126 8931 log.cc:1196] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/ef6b704e1e034a52b20b16a6a52139c6
I20260619 12:00:18.290381 8931 ts_tablet_manager.cc:1939] T ef6b704e1e034a52b20b16a6a52139c6 P 4c13ac479b9b43aa8f8a659d418462a4: Deleting consensus metadata
I20260619 12:00:18.290918 8337 catalog_manager.cc:4841] TS 4c13ac479b9b43aa8f8a659d418462a4 (127.0.203.195:45215): tablet ef6b704e1e034a52b20b16a6a52139c6 (table test-table [id=d2bf2453c2dc4b9fb1d6da2accad226e]) successfully deleted
W20260619 12:00:18.294227 8338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:46565 (error 108)
I20260619 12:00:18.294625 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 8564
I20260619 12:00:18.295573 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:18.294667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:46565 (local address 127.0.0.1:44218)
0619 12:00:18.295444 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:18.295514 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:46565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":102,"threads_started":1}
W20260619 12:00:18.296772 8338 catalog_manager.cc:4584] TS a7d0d7d11415485c804e2e3268ab737b (127.0.203.193:46565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef6b704e1e034a52b20b16a6a52139c6: Network error: Client connection negotiation failed: client connection to 127.0.203.193:46565: connect: Connection refused (error 111)
I20260619 12:00:18.302016 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 8704
I20260619 12:00:18.309557 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 8320
2026-06-19T12:00:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3131 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:18 dist-test-slave-fbq8 krb5kdc[8969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:18 dist-test-slave-fbq8 krb5kdc[8969](info): set up 2 sockets
Jun 19 12:00:18 dist-test-slave-fbq8 krb5kdc[8969](info): commencing operation
krb5kdc: starting...
W20260619 12:00:19.365536 815 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:19 dist-test-slave-fbq8 krb5kdc[8969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:00:19.500211 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34813
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:34813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:19.596539 8982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:19.596889 8982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:19.596933 8982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:19.598239 8982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:19.598313 8982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:19.598349 8982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:19.598383 8982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:19.598407 8982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:19.600147 8982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:34813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:19.600469 8982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:19.600796 8982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:19.607052 8987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:19.606938 8988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:19.607221 8982 server_base.cc:1034] running on GCE node
W20260619 12:00:19.606938 8990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:19.607635 8982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:19.608052 8982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:19.609264 8982 hybrid_clock.cc:648] HybridClock initialized: now 1781870419609252 us; error 51 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T12:00:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1306 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:19 dist-test-slave-fbq8 krb5kdc[8995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:19 dist-test-slave-fbq8 krb5kdc[8995](info): set up 2 sockets
Jun 19 12:00:19 dist-test-slave-fbq8 krb5kdc[8995](info): commencing operation
krb5kdc: starting...
W20260619 12:00:20.701897 815 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:20 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:00:20.840566 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:38415
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:38415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:20.937309 9008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:20.937597 9008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:20.937626 9008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:20.938946 9008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:20.938994 9008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:20.939009 9008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:20.939023 9008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:20.939035 9008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:20.940681 9008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:38415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:38415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:20.940979 9008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:20.941247 9008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:20.947788 9016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:20.947798 9013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:20.947767 9014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:20.947971 9008 server_base.cc:1034] running on GCE node
I20260619 12:00:20.948292 9008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:20.948750 9008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:20.949972 9008 hybrid_clock.cc:648] HybridClock initialized: now 1781870420949917 us; error 91 us; skew 500 ppm
Jun 19 12:00:20 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:20.958634 9008 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:00:20.959187 9008 webserver.cc:460] Webserver started at http://127.0.203.254:32887/ using document root <none> and password file <none>
I20260619 12:00:20.959506 9008 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:20.959568 9008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:20.959750 9008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:20.961081 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "9f3474df5cf24665a4b081bcc68a4426"
format_stamp: "Formatted at 2026-06-19 12:00:20 on dist-test-slave-fbq8"
server_key: "967784bba29545486ad407cdff1ad19c"
server_key_iv: "f9267d953ad86d048aeee4790f1ec63b"
server_key_version: "encryptionkey@0"
I20260619 12:00:20.961520 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "9f3474df5cf24665a4b081bcc68a4426"
format_stamp: "Formatted at 2026-06-19 12:00:20 on dist-test-slave-fbq8"
server_key: "967784bba29545486ad407cdff1ad19c"
server_key_iv: "f9267d953ad86d048aeee4790f1ec63b"
server_key_version: "encryptionkey@0"
I20260619 12:00:20.963258 9008 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:00:20.964270 9023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:20.964529 9008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:20.964682 9008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "9f3474df5cf24665a4b081bcc68a4426"
format_stamp: "Formatted at 2026-06-19 12:00:20 on dist-test-slave-fbq8"
server_key: "967784bba29545486ad407cdff1ad19c"
server_key_iv: "f9267d953ad86d048aeee4790f1ec63b"
server_key_version: "encryptionkey@0"
I20260619 12:00:20.964788 9008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:20.977870 9008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:20.980351 9008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:20.980500 9008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:20.985581 9008 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:38415
I20260619 12:00:20.985667 9075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:38415 every 8 connection(s)
I20260619 12:00:20.986305 9008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:20.986894 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9008
I20260619 12:00:20.987007 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:20.987254 815 external_mini_cluster.cc:1428] Setting key bc5dae9188bf6f6240fe2de7d530fbb6
I20260619 12:00:20.987390 9076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:20.991690 9076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Bootstrap starting.
I20260619 12:00:20.992525 9076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:20.992887 9076 log.cc:824] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:20.993695 9076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: No bootstrap required, opened a new log
I20260619 12:00:20.995833 9076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
Jun 19 12:00:20 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:20.996058 9076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:20.996104 9076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3474df5cf24665a4b081bcc68a4426, State: Initialized, Role: FOLLOWER
I20260619 12:00:20.996271 9076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
I20260619 12:00:20.996400 9076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:20.996447 9076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:20.996498 9076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:20.997579 9076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
I20260619 12:00:20.997730 9076 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f3474df5cf24665a4b081bcc68a4426; no voters:
I20260619 12:00:20.998071 9076 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:20.998183 9081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:20.998396 9081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 LEADER]: Becoming Leader. State: Replica: 9f3474df5cf24665a4b081bcc68a4426, State: Running, Role: LEADER
I20260619 12:00:20.998471 9076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:20.998955 9081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
I20260619 12:00:21.001104 9083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3474df5cf24665a4b081bcc68a4426. Latest consensus state: current_term: 1 leader_uuid: "9f3474df5cf24665a4b081bcc68a4426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } } }
I20260619 12:00:21.001624 9083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:21.001124 9082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f3474df5cf24665a4b081bcc68a4426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } } }
W20260619 12:00:21.001493 9096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 12:00:21.001983 9082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: This master's current role is: LEADER
W20260619 12:00:21.001992 9096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:00:21.002782 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:20.987917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.0.203.254:38415)
0619 12:00:20.988129 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:20.988138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:20.988169 (+ 31us) server_negotiation.cc:407] Connection header received
0619 12:00:20.989063 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:20.989074 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:20.989147 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:20.989294 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:20.990237 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:20.991348 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:20.992545 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:20.993285 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:20.993472 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:20.997260 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:20.997265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:20.997265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:20.997287 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:21.000408 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.001039 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.001044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.001045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.001189 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.001409 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.001410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.001411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.001788 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:21.001982 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:21.002321 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 12:00:21.002517 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260619 12:00:21.004755 9098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:21.005101 9098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:21.007627 9098 catalog_manager.cc:1347] Generated new cluster ID: bb35d59aff1b4b9e937856f35a460959
I20260619 12:00:21.007730 9098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:21.022464 9098 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:21.023185 9098 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:21.028128 9098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Generated new TSK 0
I20260619 12:00:21.028390 9098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:00:21.061879 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:38415
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:21.156992 9102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:21.157317 9102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:21.157349 9102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:21.158576 9102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:21.158620 9102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:21.158648 9102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:21.160236 9102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:21.160476 9102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:21.160720 9102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:21.166520 9107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.166625 9110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.166520 9108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:21.167038 9102 server_base.cc:1034] running on GCE node
I20260619 12:00:21.167450 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:21.167730 9102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:21.168936 9102 hybrid_clock.cc:648] HybridClock initialized: now 1781870421168909 us; error 45 us; skew 500 ppm
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:21.171996 9102 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:00:21.172537 9102 webserver.cc:460] Webserver started at http://127.0.203.193:43535/ using document root <none> and password file <none>
I20260619 12:00:21.172829 9102 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:21.172904 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:21.173067 9102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:21.174247 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "26c7ff84c15e448f9d803f56f7ff07a8"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "560aa90878a57a6e68ca4ba3da821a4b"
server_key_iv: "9e6a3d315d36229bd3ae8b8ef380d0d6"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.174655 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "26c7ff84c15e448f9d803f56f7ff07a8"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "560aa90878a57a6e68ca4ba3da821a4b"
server_key_iv: "9e6a3d315d36229bd3ae8b8ef380d0d6"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.176215 9102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:21.177271 9117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.177603 9102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:00:21.177700 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "26c7ff84c15e448f9d803f56f7ff07a8"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "560aa90878a57a6e68ca4ba3da821a4b"
server_key_iv: "9e6a3d315d36229bd3ae8b8ef380d0d6"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.177752 9102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:21.189790 9102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:21.192198 9102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:21.192322 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:21.192555 9102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:21.192965 9102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:21.193003 9102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.193030 9102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:21.193084 9102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.200853 9102 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:46413
I20260619 12:00:21.200917 9230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:46413 every 8 connection(s)
I20260619 12:00:21.201269 9102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:21.208240 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9102
I20260619 12:00:21.208395 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:21.208628 815 external_mini_cluster.cc:1428] Setting key 7c208322528f504442e06189f0a83061
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:21.213485 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:21.202156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:43977 (local address 127.0.203.254:38415)
0619 12:00:21.202237 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:21.202239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:21.202450 (+ 211us) server_negotiation.cc:407] Connection header received
0619 12:00:21.203526 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:21.203528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:21.203540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:21.203630 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:21.204441 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.205154 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.206559 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.206774 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.206843 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:21.210083 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:21.210085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:21.210085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:21.210098 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:21.211717 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.212344 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.212345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.212346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.212417 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.212731 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.212732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.212740 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.212872 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:21.212960 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:21.213262 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 12:00:21.213373 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:00:21.213744 9233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:21.202099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:43977)
0619 12:00:21.202294 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:21.202339 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:21.203419 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:21.203754 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:21.203758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:21.204074 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:21.204313 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:21.204317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.205354 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.205356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:21.206438 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:21.206442 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.206920 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.206921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:21.207070 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:21.207753 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:21.207763 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:21.209939 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:21.211907 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:21.211916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:21.211925 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:21.212210 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:21.212495 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:21.212496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:21.212496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:21.212620 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:21.212949 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:21.212955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:21.213115 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 12:00:21.213252 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 12:00:21.213445 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
I20260619 12:00:21.214363 9231 heartbeater.cc:347] Connected to a master server at 127.0.203.254:38415
I20260619 12:00:21.214488 9231 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:21.214717 9231 heartbeater.cc:511] Master 127.0.203.254:38415 requested a full tablet report, sending...
I20260619 12:00:21.215451 9040 ts_manager.cc:194] Registered new tserver with Master: 26c7ff84c15e448f9d803f56f7ff07a8 (127.0.203.193:46413)
I20260619 12:00:21.216833 9040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:43977
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:00:21.245760 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:38415
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:21.340173 9236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:21.340466 9236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:21.340503 9236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:21.341735 9236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:21.341805 9236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:21.341852 9236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:21.343495 9236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:21.343993 9236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:21.344306 9236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:00:21.350273 9236 server_base.cc:1034] running on GCE node
W20260619 12:00:21.350308 9242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.350308 9244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.350389 9241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:21.350901 9236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:21.351209 9236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:21.352425 9236 hybrid_clock.cc:648] HybridClock initialized: now 1781870421352413 us; error 48 us; skew 500 ppm
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:21.355666 9236 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:00:21.356195 9236 webserver.cc:460] Webserver started at http://127.0.203.194:40271/ using document root <none> and password file <none>
I20260619 12:00:21.356482 9236 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:21.356554 9236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:21.356769 9236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:21.357952 9236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "857e443be66a449fb983ec4f55f484b2"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "4382ba155e1b16dd44a94e85745917d3"
server_key_iv: "5d558b47ee682a4b40b90ce29dfc7c3c"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.358350 9236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "857e443be66a449fb983ec4f55f484b2"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "4382ba155e1b16dd44a94e85745917d3"
server_key_iv: "5d558b47ee682a4b40b90ce29dfc7c3c"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.359896 9236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:21.360919 9251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.361240 9236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:21.361443 9236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "857e443be66a449fb983ec4f55f484b2"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "4382ba155e1b16dd44a94e85745917d3"
server_key_iv: "5d558b47ee682a4b40b90ce29dfc7c3c"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.361580 9236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:21.375358 9236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:21.378034 9236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:21.378255 9236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:21.378517 9236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:21.378890 9236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:21.378950 9236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.378990 9236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:21.379038 9236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.386155 9236 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:39709
I20260619 12:00:21.386205 9364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:39709 every 8 connection(s)
I20260619 12:00:21.386546 9236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:21.392468 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9236
I20260619 12:00:21.392683 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:21.392910 815 external_mini_cluster.cc:1428] Setting key 69a8903f74313cf76e8364af5e733df9
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:21.398294 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:21.387343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:42659 (local address 127.0.203.254:38415)
0619 12:00:21.387410 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:21.387413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:21.387602 (+ 189us) server_negotiation.cc:407] Connection header received
0619 12:00:21.388563 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:21.388564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:21.388576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:21.388747 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:21.389433 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.390003 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.391102 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.391317 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.391407 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:21.394657 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:21.394659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:21.394659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:21.394668 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:21.396449 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.397049 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.397050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.397050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.397130 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.397416 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.397417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.397417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.397588 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:21.397668 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:21.398052 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 12:00:21.398169 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 12:00:21.398435 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:21.387289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:42659)
0619 12:00:21.387453 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:21.387498 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:21.388436 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:21.388819 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:21.388822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:21.389083 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:21.389299 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:21.389303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.390119 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.390120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:21.390998 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:21.391001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.391430 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.391431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:21.391535 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:21.392355 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:21.392367 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:21.394533 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:21.396577 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:21.396586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:21.396589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:21.396960 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:21.397219 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:21.397219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:21.397220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:21.397341 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:21.397668 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:21.397673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:21.397829 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 12:00:21.398001 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 12:00:21.398186 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260619 12:00:21.399227 9365 heartbeater.cc:347] Connected to a master server at 127.0.203.254:38415
I20260619 12:00:21.399366 9365 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:21.399593 9365 heartbeater.cc:511] Master 127.0.203.254:38415 requested a full tablet report, sending...
I20260619 12:00:21.400107 9040 ts_manager.cc:194] Registered new tserver with Master: 857e443be66a449fb983ec4f55f484b2 (127.0.203.194:39709)
I20260619 12:00:21.400712 9040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:42659
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:00:21.424513 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:38415
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:21.512450 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:21.512769 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:21.512801 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:21.514062 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:21.514132 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:21.514161 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:21.515805 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:21.516086 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:21.516368 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:21.522007 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.521962 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.522024 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:21.522099 9370 server_base.cc:1034] running on GCE node
I20260619 12:00:21.522496 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:21.522792 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:21.523967 9370 hybrid_clock.cc:648] HybridClock initialized: now 1781870421523942 us; error 46 us; skew 500 ppm
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:21.527043 9370 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:00:21.527631 9370 webserver.cc:460] Webserver started at http://127.0.203.195:41353/ using document root <none> and password file <none>
I20260619 12:00:21.527976 9370 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:21.528066 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:21.528235 9370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:21.529675 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "eff4f24c173e41abb7c9cdd08ea418cc"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "37833fbff0104f46dda2efd749f4667b"
server_key_iv: "9a056fa34f0816a3f8f45a1046983a2b"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.530100 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "eff4f24c173e41abb7c9cdd08ea418cc"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "37833fbff0104f46dda2efd749f4667b"
server_key_iv: "9a056fa34f0816a3f8f45a1046983a2b"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.531716 9370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:21.532708 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.533023 9370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:21.533146 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "eff4f24c173e41abb7c9cdd08ea418cc"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "37833fbff0104f46dda2efd749f4667b"
server_key_iv: "9a056fa34f0816a3f8f45a1046983a2b"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.533248 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:21.568886 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:21.570990 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:21.571182 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:21.571456 9370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:21.571844 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:21.571905 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.571944 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:21.571974 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.579543 9370 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:45893
I20260619 12:00:21.579591 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:45893 every 8 connection(s)
I20260619 12:00:21.579953 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:21.580745 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9370
I20260619 12:00:21.580844 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:21.581074 815 external_mini_cluster.cc:1428] Setting key 1da91595da3a656cf788c5fd63de4c51
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:21.591365 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:21.581377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:50483 (local address 127.0.203.254:38415)
0619 12:00:21.581446 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:21.581448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:21.581460 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:21.582675 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:21.582676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:21.582690 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:21.582737 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:21.583469 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.584031 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.585124 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.585348 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.585437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:21.588012 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:21.588014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:21.588015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:21.588026 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:21.589697 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.590170 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.590172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.590172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.590252 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:21.590528 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:21.590529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:21.590530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:21.590757 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:21.590852 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:21.591117 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 12:00:21.591231 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 12:00:21.591392 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:21.580979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:50483)
0619 12:00:21.581274 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:21.581328 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:21.582559 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:21.582814 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:21.582819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:21.583139 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:21.583366 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:21.583370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.584130 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.584131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:21.585019 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:21.585021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:21.585446 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:21.585447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:21.585594 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:21.586204 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:21.586213 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:21.587846 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:21.589801 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:21.589809 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:21.589812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:21.590080 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:21.590342 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:21.590342 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:21.590342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:21.590447 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:21.590843 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:21.590847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:21.590973 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 12:00:21.591062 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:00:21.591194 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":116,"threads_started":1}
I20260619 12:00:21.592070 9499 heartbeater.cc:347] Connected to a master server at 127.0.203.254:38415
I20260619 12:00:21.592195 9499 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:21.592415 9499 heartbeater.cc:511] Master 127.0.203.254:38415 requested a full tablet report, sending...
I20260619 12:00:21.592898 9040 ts_manager.cc:194] Registered new tserver with Master: eff4f24c173e41abb7c9cdd08ea418cc (127.0.203.195:45893)
I20260619 12:00:21.593426 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:00:21.593514 9040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:50483
WARNING: no policy specified for oryx/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab.
Entry for principal oryx/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab.
WARNING: no policy specified for oryx/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab.
Entry for principal oryx/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab.
WARNING: no policy specified for oryx/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab.
Entry for principal oryx/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab.
I20260619 12:00:21.680538 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9102
I20260619 12:00:21.685999 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9236
I20260619 12:00:21.690975 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9370
I20260619 12:00:21.697073 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9008
I20260619 12:00:21.703330 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:38415
--webserver_interface=127.0.203.254
--webserver_port=32887
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:38415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:21.788880 9508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:21.789151 9508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:21.789207 9508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:21.790400 9508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:21.790484 9508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:21.790512 9508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:21.790545 9508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:21.790568 9508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:21.792141 9508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:38415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:38415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:21.792429 9508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:21.792724 9508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:21.798163 9516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.798183 9514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:21.798276 9508 server_base.cc:1034] running on GCE node
W20260619 12:00:21.798369 9513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:21.798756 9508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:21.799155 9508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:21.800359 9508 hybrid_clock.cc:648] HybridClock initialized: now 1781870421800337 us; error 43 us; skew 500 ppm
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:21.803550 9508 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:00:21.804054 9508 webserver.cc:460] Webserver started at http://127.0.203.254:32887/ using document root <none> and password file <none>
I20260619 12:00:21.804296 9508 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:21.804382 9508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:21.805967 9508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:21.806821 9523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.807138 9508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:21.807286 9508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "9f3474df5cf24665a4b081bcc68a4426"
format_stamp: "Formatted at 2026-06-19 12:00:20 on dist-test-slave-fbq8"
server_key: "967784bba29545486ad407cdff1ad19c"
server_key_iv: "f9267d953ad86d048aeee4790f1ec63b"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.807799 9508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:21.833338 9508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:21.835906 9508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:21.836090 9508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:21.841331 9508 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:38415
I20260619 12:00:21.841691 9575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:38415 every 8 connection(s)
I20260619 12:00:21.841779 9508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:21.843164 9576 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:00:21.843799 9576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Bootstrap starting.
I20260619 12:00:21.850263 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9508
I20260619 12:00:21.850615 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:46413
--local_ip_for_outbound_sockets=127.0.203.193
--tserver_master_addrs=127.0.203.254:38415
--webserver_port=43535
--webserver_interface=127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:00:21.852576 9576 log.cc:824] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:21.854315 9576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:00:21.854627 9576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Bootstrap complete.
I20260619 12:00:21.857026 9576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
I20260619 12:00:21.857254 9576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3474df5cf24665a4b081bcc68a4426, State: Initialized, Role: FOLLOWER
I20260619 12:00:21.857427 9576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
I20260619 12:00:21.857524 9576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:21.857571 9576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:21.857630 9576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:00:21.858664 9576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
I20260619 12:00:21.858795 9576 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f3474df5cf24665a4b081bcc68a4426; no voters:
I20260619 12:00:21.859012 9576 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:00:21.859048 9580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:00:21.859157 9580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [term 2 LEADER]: Becoming Leader. State: Replica: 9f3474df5cf24665a4b081bcc68a4426, State: Running, Role: LEADER
I20260619 12:00:21.859314 9576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:21.859301 9580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } }
I20260619 12:00:21.859880 9582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3474df5cf24665a4b081bcc68a4426. Latest consensus state: current_term: 2 leader_uuid: "9f3474df5cf24665a4b081bcc68a4426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } } }
I20260619 12:00:21.860064 9582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:21.860309 9581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f3474df5cf24665a4b081bcc68a4426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3474df5cf24665a4b081bcc68a4426" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 38415 } } }
I20260619 12:00:21.860479 9581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:21.860695 9590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:21.861472 9590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:21.861637 9590 catalog_manager.cc:1259] Loaded cluster ID: bb35d59aff1b4b9e937856f35a460959
I20260619 12:00:21.861686 9590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:21.863092 9590 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:21.863291 9590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f3474df5cf24665a4b081bcc68a4426: Loaded TSK: 0
I20260619 12:00:21.863528 9590 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:00:21.943007 9578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:21.943423 9578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:21.943475 9578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:21.946072 9578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:21.946134 9578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:21.946168 9578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:21.948252 9578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.193.keytab
--principal=oryx/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:46413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=43535
--tserver_master_addrs=127.0.203.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:21.948612 9578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:21.948944 9578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:21.955332 9602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.955320 9605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:21.955363 9603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:21.955715 9578 server_base.cc:1034] running on GCE node
I20260619 12:00:21.955940 9578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:21.956220 9578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:21.957401 9578 hybrid_clock.cc:648] HybridClock initialized: now 1781870421957368 us; error 58 us; skew 500 ppm
Jun 19 12:00:21 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870421, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:21.960253 9578 init.cc:377] Logged in from keytab as oryx/127.0.203.193@KRBTEST.COM (short username oryx)
I20260619 12:00:21.960947 9578 webserver.cc:460] Webserver started at http://127.0.203.193:43535/ using document root <none> and password file <none>
I20260619 12:00:21.961256 9578 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:21.961324 9578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:21.963480 9578 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:21.964249 9612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.964541 9578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:21.964671 9578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "26c7ff84c15e448f9d803f56f7ff07a8"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "560aa90878a57a6e68ca4ba3da821a4b"
server_key_iv: "9e6a3d315d36229bd3ae8b8ef380d0d6"
server_key_version: "encryptionkey@0"
I20260619 12:00:21.965139 9578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:21.980777 9578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:21.985101 9578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:21.985339 9578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:21.985605 9578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:21.986110 9578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:21.986189 9578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.986248 9578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:21.986281 9578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:21.994539 9578 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:46413
I20260619 12:00:21.994676 9725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:46413 every 8 connection(s)
I20260619 12:00:21.995042 9578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:21.996887 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9578
I20260619 12:00:21.997238 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:39709
--local_ip_for_outbound_sockets=127.0.203.194
--tserver_master_addrs=127.0.203.254:38415
--webserver_port=40271
--webserver_interface=127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.008249 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:21.996096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:53933)
0619 12:00:21.996409 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:21.996466 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:21.997176 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:21.997776 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:21.997781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:21.998783 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:21.999119 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:21.999126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.001001 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.001004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.003128 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.003134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.004388 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.004390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.004536 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.005377 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.005388 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.008053 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":163,"threads_started":1}
W20260619 12:00:22.008569 9729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:21.996194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:53933 (local address 127.0.203.254:38415)
0619 12:00:21.996445 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:21.996456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:21.996604 (+ 148us) server_negotiation.cc:407] Connection header received
0619 12:00:21.997445 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:21.997453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:21.997516 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:21.997694 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:21.999335 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.000858 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.003310 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.004156 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.004340 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.008301 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:53933: BlockingRecv error: recv got EOF from 127.0.203.193:53933 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
W20260619 12:00:22.008626 9726 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:38415 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.203.254:38415: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.014859 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.008937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:45673)
0619 12:00:22.008984 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.009002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.009065 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.009284 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.009284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.009476 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.009556 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.009559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.010264 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.010265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.011230 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.011232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.011783 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.011784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.011857 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.012443 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.012449 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.014770 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26,"thread_start_us":116,"threads_started":1}
W20260619 12:00:22.015115 9729 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.009005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:45673 (local address 127.0.203.254:38415)
0619 12:00:22.009061 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.009063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.009074 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:22.009160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.009161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.009185 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.009243 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.009651 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.010165 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.011415 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.011698 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.011765 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.015024 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:45673: BlockingRecv error: recv got EOF from 127.0.203.193:45673 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.020939 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.015291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:52171)
0619 12:00:22.015344 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.015357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.015404 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.015763 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.015765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.015918 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.015959 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.015961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.017032 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.017033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.017726 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.017728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.018143 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.018144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.018202 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.018665 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.018670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.020845 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 12:00:22.021015 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.015374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:52171 (local address 127.0.203.254:38415)
0619 12:00:22.015602 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.015604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.015612 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:22.015638 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.015643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.015655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.015720 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.016074 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.016938 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.017823 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.018065 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.018145 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.020926 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:52171: BlockingRecv error: recv got EOF from 127.0.203.193:52171 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":124,"threads_started":1}
W20260619 12:00:22.021137 9726 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:00:22.089378 9730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:22.089600 9730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:22.089624 9730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:22.090762 9730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:22.090821 9730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:22.090848 9730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:22.092242 9730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.194.keytab
--principal=oryx/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:39709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=40271
--tserver_master_addrs=127.0.203.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:22.092442 9730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:22.092622 9730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:22.098645 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:22.098671 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:22.098681 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:22.099102 9730 server_base.cc:1034] running on GCE node
I20260619 12:00:22.099422 9730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:22.099822 9730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:22.101032 9730 hybrid_clock.cc:648] HybridClock initialized: now 1781870422100999 us; error 61 us; skew 500 ppm
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870422, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:22.104020 9730 init.cc:377] Logged in from keytab as oryx/127.0.203.194@KRBTEST.COM (short username oryx)
I20260619 12:00:22.104570 9730 webserver.cc:460] Webserver started at http://127.0.203.194:40271/ using document root <none> and password file <none>
I20260619 12:00:22.104862 9730 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:22.104951 9730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:22.106527 9730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:22.107295 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:22.107661 9730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:22.107823 9730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "857e443be66a449fb983ec4f55f484b2"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "4382ba155e1b16dd44a94e85745917d3"
server_key_iv: "5d558b47ee682a4b40b90ce29dfc7c3c"
server_key_version: "encryptionkey@0"
I20260619 12:00:22.108218 9730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:22.120868 9730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:22.123148 9730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:22.123346 9730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:22.123613 9730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:22.123950 9730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:22.124009 9730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:22.124047 9730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:22.124092 9730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:22.131305 9730 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:39709
I20260619 12:00:22.131350 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:39709 every 8 connection(s)
I20260619 12:00:22.131737 9730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:22.135270 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9730
I20260619 12:00:22.135632 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:45893
--local_ip_for_outbound_sockets=127.0.203.195
--tserver_master_addrs=127.0.203.254:38415
--webserver_port=41353
--webserver_interface=127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:00:22.141474 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.132688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:41081 (local address 127.0.203.254:38415)
0619 12:00:22.132782 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.132785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.133034 (+ 249us) server_negotiation.cc:407] Connection header received
0619 12:00:22.133686 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.133687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.133705 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.133761 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.134853 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.135639 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.137072 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.137321 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.137418 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.141415 (+ 3997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:41081: BlockingRecv error: recv got EOF from 127.0.203.194:41081 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 12:00:22.141592 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.132585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:41081)
0619 12:00:22.132878 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.132925 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.133558 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.133853 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.133857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.134509 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.134739 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.134743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.135809 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.135812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.136928 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.136932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.137425 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.137426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.137569 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.138201 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.138210 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.141277 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":145,"threads_started":1}
W20260619 12:00:22.141964 9861 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:38415 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.203.254:38415: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.148316 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.142274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:54121)
0619 12:00:22.142335 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.142348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.142395 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.142588 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.142589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.142749 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.142810 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.142812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.143874 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.143876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.144426 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.144427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.145000 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.145001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.145070 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.145742 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.145747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.148227 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 12:00:22.148594 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.142348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:54121 (local address 127.0.203.254:38415)
0619 12:00:22.142409 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.142412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.142424 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:22.142455 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.142456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.142473 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.142555 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.142905 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.143765 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.144523 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.144902 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.144984 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.148511 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:54121: BlockingRecv error: recv got EOF from 127.0.203.194:54121 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.155244 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.148718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:60145)
0619 12:00:22.148841 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.148853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.148898 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.149088 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.149089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.149249 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.149291 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.149292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.150846 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.150848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.151746 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.151748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.152245 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.152246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.152306 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.152928 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.152933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.155140 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 12:00:22.155282 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.148860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:60145 (local address 127.0.203.254:38415)
0619 12:00:22.148957 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.148960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.148968 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:22.149003 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.149004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.149015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.149084 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.149645 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.150471 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.151918 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.152137 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.152233 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.155221 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:60145: BlockingRecv error: recv got EOF from 127.0.203.194:60145 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 12:00:22.155401 9861 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:00:22.228998 9864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:22.229389 9864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:22.229465 9864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:22.230751 9864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:22.230829 9864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:22.230892 9864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:22.232571 9864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/krb5kdc/oryx_127.0.203.195.keytab
--principal=oryx/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=41353
--tserver_master_addrs=127.0.203.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:22.232890 9864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:22.233150 9864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:22.238786 9869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:22.238806 9870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:22.239004 9864 server_base.cc:1034] running on GCE node
W20260619 12:00:22.238788 9872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:22.239279 9864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:22.239496 9864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:22.240720 9864 hybrid_clock.cc:648] HybridClock initialized: now 1781870422240680 us; error 58 us; skew 500 ppm
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870422, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:22.243912 9864 init.cc:377] Logged in from keytab as oryx/127.0.203.195@KRBTEST.COM (short username oryx)
I20260619 12:00:22.244517 9864 webserver.cc:460] Webserver started at http://127.0.203.195:41353/ using document root <none> and password file <none>
I20260619 12:00:22.244833 9864 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:22.244923 9864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:22.246582 9864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:22.247439 9879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:22.247733 9864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:22.247891 9864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "eff4f24c173e41abb7c9cdd08ea418cc"
format_stamp: "Formatted at 2026-06-19 12:00:21 on dist-test-slave-fbq8"
server_key: "37833fbff0104f46dda2efd749f4667b"
server_key_iv: "9a056fa34f0816a3f8f45a1046983a2b"
server_key_version: "encryptionkey@0"
I20260619 12:00:22.248415 9864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:22.263780 9864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:22.266422 9864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:22.266613 9864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:22.266891 9864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:22.267236 9864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:22.267293 9864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:22.267344 9864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:22.267369 9864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:22.275256 9864 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:45893
I20260619 12:00:22.275305 9992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:45893 every 8 connection(s)
I20260619 12:00:22.275701 9864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:22.282182 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 9864
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.284690 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.276547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:38459)
0619 12:00:22.276771 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.276817 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.277378 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.277681 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.277684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.278294 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.278556 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.278561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.279334 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.279335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.280218 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.280219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.280796 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.280797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.280914 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.281473 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.281483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.284444 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":140,"threads_started":1}
W20260619 12:00:22.284704 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.276567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:38459 (local address 127.0.203.254:38415)
0619 12:00:22.276698 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.276700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.276968 (+ 268us) server_negotiation.cc:407] Connection header received
0619 12:00:22.277495 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.277496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.277509 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.277597 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.278667 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.279229 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.280334 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.280529 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.280845 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.284569 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:38459: BlockingRecv error: recv got EOF from 127.0.203.195:38459 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 12:00:22.285005 9993 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:38415 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.203.254:38415: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.291908 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.285247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:52203)
0619 12:00:22.285344 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.285362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.285426 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.285575 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.285576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.285749 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.285816 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.285819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.286617 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.286618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.287513 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.287516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.288140 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.288141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.288231 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.289070 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.289076 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.291794 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 12:00:22.292006 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.282945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.0.203.254:38415)
0619 12:00:22.283008 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.283011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.283019 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:22.283045 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.283046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.283061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.283113 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.283957 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.284753 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.285829 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.286049 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.286130 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.287696 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:22.287702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:22.287703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:22.287721 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:22.290229 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:22.290755 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:22.290759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:22.290759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:22.290878 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:22.291107 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:22.291108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:22.291109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:22.291409 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:22.291577 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:22.291729 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 12:00:22.291883 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 12:00:22.292135 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.285378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:52203 (local address 127.0.203.254:38415)
0619 12:00:22.285417 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.285420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.285428 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:00:22.285498 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.285499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.285509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.285565 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.285908 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.286494 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.287641 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.287988 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.288075 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.292062 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:52203: BlockingRecv error: recv got EOF from 127.0.203.195:52203 (error 108)
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":20}
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:22 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:22.297359 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:22.292229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:34507)
0619 12:00:22.292406 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:22.292418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:22.292463 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:22.292790 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:22.292792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:22.292969 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:22.293009 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.293011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.293816 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.293817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.294387 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:22.294388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.294785 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.294786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:22.294835 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.295336 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:22.295341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:22.297197 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
W20260619 12:00:22.297392 9732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:22.292547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:34507 (local address 127.0.203.254:38415)
0619 12:00:22.292616 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:22.292618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:22.292628 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:22.292672 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:22.292673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:22.292684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:22.292749 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:22.293094 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.293714 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.294463 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:22.294664 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:22.294744 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:22.297337 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:34507: BlockingRecv error: recv got EOF from 127.0.203.195:34507 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 12:00:22.297559 9993 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 12:00:23 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:23 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:23.027901 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:23.021708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:35263)
0619 12:00:23.021947 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:23.021970 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:23.022092 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:23.022379 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:23.022381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:23.022587 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:23.022685 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:23.022686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.023713 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.023714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:23.024356 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:23.024358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.024879 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.024880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:23.024944 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:23.025540 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:23.025546 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:23.027756 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
W20260619 12:00:23.027925 9999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:23.021920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:35263 (local address 127.0.203.254:38415)
0619 12:00:23.022144 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:23.022146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:23.022159 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:00:23.022189 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:23.022190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:23.022205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:23.022292 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:23.022823 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.023535 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.024512 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.024761 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.024853 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:23.027856 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:35263: BlockingRecv error: recv got EOF from 127.0.203.193:35263 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":129,"threads_started":1}
Jun 19 12:00:23 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:23 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:23.161751 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:23.155887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:38629)
0619 12:00:23.156162 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:23.156179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:23.156271 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:23.156441 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:23.156442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:23.156681 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:23.156732 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:23.156735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.157422 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.157423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:23.157963 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:23.157965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.158331 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.158332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:23.158385 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:23.158904 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:23.158910 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:23.161665 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":119,"threads_started":1}
W20260619 12:00:23.161939 9999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:23.156049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:38629 (local address 127.0.203.254:38415)
0619 12:00:23.156167 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:23.156170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:23.156247 (+ 77us) server_negotiation.cc:407] Connection header received
0619 12:00:23.156344 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:23.156345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:23.156358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:23.156435 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:23.156821 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.157324 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.158050 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.158242 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.158331 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:23.161833 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:38629: BlockingRecv error: recv got EOF from 127.0.203.194:38629 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 12:00:23 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:23 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:23.304193 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:23.298190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:33781)
0619 12:00:23.298424 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:23.298440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:23.298537 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:23.298706 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:23.298707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:23.298950 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:23.299010 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:23.299013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.299975 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.299976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:23.300615 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:23.300617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.301073 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.301074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:23.301133 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:23.301731 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:23.301736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:23.304096 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":120,"threads_started":1}
W20260619 12:00:23.304399 9999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:23.298293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:33781 (local address 127.0.203.254:38415)
0619 12:00:23.298375 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:23.298377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:23.298509 (+ 132us) server_negotiation.cc:407] Connection header received
0619 12:00:23.298632 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:23.298633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:23.298649 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:23.298700 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:23.299107 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.299869 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.300772 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:23.300984 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:23.301072 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:23.304277 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:33781: BlockingRecv error: recv got EOF from 127.0.203.195:33781 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 12:00:24 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:24 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:24.034420 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:24.028537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:39341)
0619 12:00:24.028896 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:24.028917 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:24.029029 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:24.029225 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:24.029226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:24.029438 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:24.029545 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:24.029547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.030401 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.030402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:24.031000 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:24.031002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.031441 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.031442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:24.031516 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:24.032126 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:24.032131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:24.034344 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":173,"threads_started":1}
W20260619 12:00:24.034608 10003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:24.028719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:39341 (local address 127.0.203.254:38415)
0619 12:00:24.029057 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:24.029059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:24.029069 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:24.029098 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:24.029099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:24.029112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:24.029186 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:24.029659 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.030270 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.031109 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.031344 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.031410 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:24.034496 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:39341: BlockingRecv error: recv got EOF from 127.0.203.193:39341 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":216,"threads_started":1}
Jun 19 12:00:24 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:24 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:24.168377 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:24.162311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:41819)
0619 12:00:24.162554 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:24.162572 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:24.162721 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:24.162945 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:24.162946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:24.163217 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:24.163298 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:24.163301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.164410 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.164411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:24.165020 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:24.165021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.165482 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.165483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:24.165536 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:24.166133 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:24.166138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:24.168309 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":111,"threads_started":1}
W20260619 12:00:24.168519 10003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:24.162444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:41819 (local address 127.0.203.254:38415)
0619 12:00:24.162559 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:24.162562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:24.162707 (+ 145us) server_negotiation.cc:407] Connection header received
0619 12:00:24.162828 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:24.162829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:24.162846 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:24.162915 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:24.163407 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.164255 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.165141 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.165373 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.165459 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:24.168426 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:41819: BlockingRecv error: recv got EOF from 127.0.203.194:41819 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 12:00:24 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:24 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:24.310355 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:24.304891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:50713)
0619 12:00:24.305083 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:24.305095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:24.305193 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:24.305348 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:24.305349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:24.305555 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:24.305614 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:24.305615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.306333 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.306334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:24.306909 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:24.306910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.307288 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.307289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:24.307353 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:24.307930 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:24.307935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:24.310268 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":116,"threads_started":1}
W20260619 12:00:24.310473 10003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:24.305083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:50713 (local address 127.0.203.254:38415)
0619 12:00:24.305178 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:24.305181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:24.305194 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:00:24.305268 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:24.305269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:24.305288 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:24.305343 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:24.305702 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.306245 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.306993 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:24.307201 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:24.307287 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:24.310374 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:50713: BlockingRecv error: recv got EOF from 127.0.203.195:50713 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 12:00:25 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:25 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:25.040951 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:25.035148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:42991)
0619 12:00:25.035322 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:25.035335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:25.035401 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:25.035654 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:25.035655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:25.035851 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:25.035920 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:25.035921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.036797 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.036799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:25.037409 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:25.037411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.037882 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.037883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:25.037937 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:25.038569 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:25.038574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:25.040866 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
W20260619 12:00:25.041105 10007 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:25.035220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:42991 (local address 127.0.203.254:38415)
0619 12:00:25.035370 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:25.035373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:25.035388 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:00:25.035534 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:25.035535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:25.035550 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:25.035629 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:25.036000 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.036545 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.037514 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.037752 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.037825 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:25.040985 (+ 3160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:42991: BlockingRecv error: recv got EOF from 127.0.203.193:42991 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
Jun 19 12:00:25 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:25 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:25.174170 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:25.169058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:54071)
0619 12:00:25.169246 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:25.169258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:25.169336 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:25.169462 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:25.169463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:25.169655 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:25.169720 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:25.169721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.170465 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.170466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:25.171053 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:25.171054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.171460 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.171461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:25.171523 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:25.172013 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:25.172018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:25.174088 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":128,"threads_started":1}
W20260619 12:00:25.174289 10007 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:25.169107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:54071 (local address 127.0.203.254:38415)
0619 12:00:25.169169 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:25.169171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:25.169344 (+ 173us) server_negotiation.cc:407] Connection header received
0619 12:00:25.169381 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:25.169382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:25.169394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:25.169453 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:25.169827 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.170359 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.171163 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.171354 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.171426 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:25.174197 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:54071: BlockingRecv error: recv got EOF from 127.0.203.194:54071 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 12:00:25 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:25 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:25.317011 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:25.310995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:49813)
0619 12:00:25.311190 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:25.311208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:25.311277 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:25.311539 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:25.311540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:25.311805 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:25.311870 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:25.311873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.312763 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.312764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:25.313649 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:25.313652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.314061 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.314063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:25.314115 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:25.314667 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:25.314673 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:25.316925 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":135,"threads_started":1}
W20260619 12:00:25.317170 10007 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:25.311131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:49813 (local address 127.0.203.254:38415)
0619 12:00:25.311197 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:25.311200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:25.311278 (+ 78us) server_negotiation.cc:407] Connection header received
0619 12:00:25.311336 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:25.311337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:25.311349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:25.311427 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:25.312007 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.312602 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.313733 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:25.313954 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:25.314018 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:25.317071 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:49813: BlockingRecv error: recv got EOF from 127.0.203.195:49813 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 12:00:26 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:26 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:26.051875 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:26.041578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:57183)
0619 12:00:26.041770 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:26.041781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:26.041845 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:26.042239 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:26.042241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:26.042422 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:26.042476 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:26.042478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.043319 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.043319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:26.044004 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:26.044007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.044456 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.044458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:26.044523 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:26.049505 (+ 4982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:26.049512 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:26.051784 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
W20260619 12:00:26.052100 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:26.041834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:57183 (local address 127.0.203.254:38415)
0619 12:00:26.042001 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:26.042003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:26.042015 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:26.042047 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:26.042048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:26.042060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:26.042154 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:26.042607 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.043218 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.044115 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.044314 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.044407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:26.051956 (+ 7549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:57183: BlockingRecv error: recv got EOF from 127.0.203.193:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
Jun 19 12:00:26 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:26 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:26.180125 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:26.174770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:52601)
0619 12:00:26.174937 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:26.174948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:26.175011 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:26.175228 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:26.175229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:26.175392 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:26.175444 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:26.175446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.176190 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.176192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:26.176838 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:26.176840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.177222 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.177223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:26.177262 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:26.177773 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:26.177778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:26.180047 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
W20260619 12:00:26.180245 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:26.174922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:52601 (local address 127.0.203.254:38415)
0619 12:00:26.175050 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:26.175052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:26.175073 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:00:26.175109 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:26.175110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:26.175122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:26.175204 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:26.175532 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.176094 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.176947 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.177145 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.177209 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:26.180142 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:52601: BlockingRecv error: recv got EOF from 127.0.203.194:52601 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 12:00:26 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:26 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:26.322929 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:26.317663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:37113)
0619 12:00:26.317880 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:26.317897 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:26.317988 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:26.318237 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:26.318239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:26.318439 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:26.318491 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:26.318493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.319203 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.319204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:26.319832 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:26.319834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.320283 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.320284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:26.320327 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:26.320976 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:26.320981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:26.322851 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":131,"threads_started":1}
W20260619 12:00:26.323117 10011 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:26.317776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:37113 (local address 127.0.203.254:38415)
0619 12:00:26.317863 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:26.317866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:26.317976 (+ 110us) server_negotiation.cc:407] Connection header received
0619 12:00:26.318045 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:26.318046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:26.318059 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:26.318136 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:26.318591 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.319094 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.319966 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:26.320199 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:26.320275 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:26.323008 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:37113: BlockingRecv error: recv got EOF from 127.0.203.195:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 12:00:27 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:27 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:27.057902 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:27.052566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:39569)
0619 12:00:27.052776 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:27.052795 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:27.052877 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:27.053163 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:27.053164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:27.053349 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:27.053406 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:27.053408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.054210 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.054211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:27.054791 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:27.054793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.055222 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.055223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:27.055267 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:27.055969 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:27.055974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:27.057836 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
W20260619 12:00:27.058115 10015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:27.052749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:39569 (local address 127.0.203.254:38415)
0619 12:00:27.052961 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:27.052963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:27.052973 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:27.053002 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:27.053003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:27.053015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:27.053100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:27.053526 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.054088 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.054919 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.055133 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.055201 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:27.058009 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:39569: BlockingRecv error: recv got EOF from 127.0.203.193:39569 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
Jun 19 12:00:27 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:27 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:27.186197 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:27.180820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:47191)
0619 12:00:27.180984 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:27.180998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:27.181066 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:27.181332 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:27.181333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:27.181529 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:27.181575 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:27.181576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.182322 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.182323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:27.182883 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:27.182884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.183270 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.183272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:27.183335 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:27.183882 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:27.183887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:27.186110 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
W20260619 12:00:27.186322 10015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:27.181076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:47191 (local address 127.0.203.254:38415)
0619 12:00:27.181198 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:27.181200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:27.181206 (+ 6us) server_negotiation.cc:407] Connection header received
0619 12:00:27.181233 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:27.181234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:27.181244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:27.181326 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:27.181708 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.182236 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.182976 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.183176 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.183262 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:27.186250 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:47191: BlockingRecv error: recv got EOF from 127.0.203.194:47191 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 12:00:27 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:27 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:27.328737 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:27.323473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:50449)
0619 12:00:27.323712 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:27.323726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:27.323800 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:27.324046 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:27.324046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:27.324248 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:27.324315 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:27.324318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.325040 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.325041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:27.325617 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:27.325618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.326006 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.326008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:27.326074 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:27.326652 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:27.326664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:27.328596 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":124,"threads_started":1}
W20260619 12:00:27.328897 10015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:27.323544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:50449 (local address 127.0.203.254:38415)
0619 12:00:27.323621 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:27.323624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:27.323803 (+ 179us) server_negotiation.cc:407] Connection header received
0619 12:00:27.323868 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:27.323869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:27.323884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:27.324039 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:27.324393 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.324951 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.325706 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:27.325909 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:27.325974 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:27.328764 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:50449: BlockingRecv error: recv got EOF from 127.0.203.195:50449 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 12:00:28 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:28 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:28.064380 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:28.058477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:52593)
0619 12:00:28.058811 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:28.058824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:28.058987 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:28.059182 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:28.059183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:28.059424 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:28.059486 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:28.059488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.060339 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.060339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:28.061044 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:28.061046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.061402 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.061403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:28.061444 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:28.061963 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:28.061968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:28.064275 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":246,"threads_started":1}
W20260619 12:00:28.064447 10019 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:28.058677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:52593 (local address 127.0.203.254:38415)
0619 12:00:28.058796 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:28.058799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:28.058953 (+ 154us) server_negotiation.cc:407] Connection header received
0619 12:00:28.059072 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:28.059073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:28.059085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:28.059159 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:28.059575 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.060228 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.061122 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.061330 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.061392 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:28.064362 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:52593: BlockingRecv error: recv got EOF from 127.0.203.193:52593 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
Jun 19 12:00:28 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:28 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:28.192965 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:28.187078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:58443)
0619 12:00:28.187364 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:28.187378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:28.187527 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:28.187693 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:28.187694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:28.188056 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:28.188128 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:28.188132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.189136 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.189137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:28.189751 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:28.189753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.190151 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.190152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:28.190205 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:28.190793 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:28.190798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:28.192849 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":88,"threads_started":1}
W20260619 12:00:28.193009 10019 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:28.187341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:58443 (local address 127.0.203.254:38415)
0619 12:00:28.187387 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:28.187390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:28.187514 (+ 124us) server_negotiation.cc:407] Connection header received
0619 12:00:28.187591 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:28.187592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:28.187605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:28.187687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:28.188267 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.189015 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.189860 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.190063 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.190152 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:28.192939 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:58443: BlockingRecv error: recv got EOF from 127.0.203.194:58443 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 19 12:00:28 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:28 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:28.335420 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:28.329443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:51409)
0619 12:00:28.329652 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:28.329666 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:28.329769 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:28.330073 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:28.330075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:28.330298 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:28.330366 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:28.330368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.331164 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.331165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:28.331779 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:28.331781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.332220 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.332222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:28.332290 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:28.332997 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:28.333036 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:28.335314 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":151,"threads_started":1}
W20260619 12:00:28.335477 10019 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:28.329506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:51409 (local address 127.0.203.254:38415)
0619 12:00:28.329584 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:28.329586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:28.329745 (+ 159us) server_negotiation.cc:407] Connection header received
0619 12:00:28.329833 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:28.329834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:28.329848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:28.329944 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:28.330500 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.331063 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.331892 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:28.332088 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:28.332159 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:28.335408 (+ 3249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:51409: BlockingRecv error: recv got EOF from 127.0.203.195:51409 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 12:00:29 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:29 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:29.070801 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:29.065073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:37507)
0619 12:00:29.065443 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:29.065459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:29.065542 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:29.065739 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:29.065740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:29.065921 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:29.065975 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:29.065977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.066753 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.066754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:29.067345 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:29.067346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.067773 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.067774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:29.067833 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:29.068481 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:29.068485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:29.070724 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":93,"threads_started":1}
W20260619 12:00:29.071038 10023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:29.065309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:37507 (local address 127.0.203.254:38415)
0619 12:00:29.065477 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:29.065479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:29.065524 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:00:29.065597 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:29.065597 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:29.065610 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:29.065679 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:29.066100 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.066653 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.067459 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.067677 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.067740 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:29.070938 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:37507: BlockingRecv error: recv got EOF from 127.0.203.193:37507 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
Jun 19 12:00:29 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:29 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:29.199168 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:29.193725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:39361)
0619 12:00:29.193893 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:29.193913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:29.193964 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:29.194171 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:29.194172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:29.194333 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:29.194387 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:29.194388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.195111 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.195111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:29.195654 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:29.195655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.196040 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.196041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:29.196082 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:29.196610 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:29.196615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:29.199103 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
W20260619 12:00:29.199378 10023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:29.193935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:39361 (local address 127.0.203.254:38415)
0619 12:00:29.194017 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:29.194019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:29.194028 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:00:29.194059 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:29.194060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:29.194072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:29.194145 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:29.194472 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.195034 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.195766 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.195961 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.196030 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:29.199288 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:39361: BlockingRecv error: recv got EOF from 127.0.203.194:39361 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 12:00:29 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:29 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:29.341190 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:29.335990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:38623)
0619 12:00:29.336235 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:29.336249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:29.336354 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:29.336538 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:29.336539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:29.336784 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:29.336844 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:29.336846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.337569 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.337570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:29.338127 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:29.338128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.338527 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.338528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:29.338587 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:29.339150 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:29.339156 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:29.341084 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
W20260619 12:00:29.341248 10023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:29.336113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:38623 (local address 127.0.203.254:38415)
0619 12:00:29.336204 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:29.336206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:29.336331 (+ 125us) server_negotiation.cc:407] Connection header received
0619 12:00:29.336439 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:29.336440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:29.336460 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:29.336531 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:29.336930 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.337476 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.338213 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:29.338419 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:29.338516 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:29.341163 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:38623: BlockingRecv error: recv got EOF from 127.0.203.195:38623 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 12:00:30 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:30 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:30.076921 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:30.071387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:52537)
0619 12:00:30.071547 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:30.071557 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:30.071607 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:30.071894 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:30.071895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:30.072054 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:30.072104 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:30.072106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.072982 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.072983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:30.073668 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:30.073670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.074097 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.074099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:30.074177 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:30.074728 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:30.074733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:30.076857 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
W20260619 12:00:30.077405 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:30.071514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:52537 (local address 127.0.203.254:38415)
0619 12:00:30.071705 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:30.071707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:30.071720 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:00:30.071752 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:30.071752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:30.071764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:30.071867 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:30.072221 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.072871 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.073756 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.073980 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.074091 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:30.077320 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:52537: BlockingRecv error: recv got EOF from 127.0.203.193:52537 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
Jun 19 12:00:30 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:30 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:30.205808 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:30.199822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:34895)
0619 12:00:30.200106 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:30.200120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:30.200228 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:30.200386 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:30.200387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:30.200573 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:30.200622 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:30.200624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.201669 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.201670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:30.202376 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:30.202378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.202759 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.202760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:30.202802 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:30.203357 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:30.203362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:30.205686 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":157,"threads_started":1}
W20260619 12:00:30.206043 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:30.199970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:34895 (local address 127.0.203.254:38415)
0619 12:00:30.200082 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:30.200084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:30.200205 (+ 121us) server_negotiation.cc:407] Connection header received
0619 12:00:30.200290 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:30.200291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:30.200305 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:30.200380 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:30.200812 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.201559 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.202482 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.202683 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.202742 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:30.205832 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:34895: BlockingRecv error: recv got EOF from 127.0.203.194:34895 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 12:00:30 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:30 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:30.347193 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:30.341944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:54479)
0619 12:00:30.342131 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:30.342146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:30.342240 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:30.342427 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:30.342428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:30.342711 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:30.342779 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:30.342781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.343469 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.343470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:30.344096 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:30.344098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.344507 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.344507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:30.344576 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:30.345229 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:30.345235 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:30.347122 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
W20260619 12:00:30.347378 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:30.342017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:54479 (local address 127.0.203.254:38415)
0619 12:00:30.342079 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:30.342081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:30.342252 (+ 171us) server_negotiation.cc:407] Connection header received
0619 12:00:30.342308 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:30.342309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:30.342326 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:30.342395 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:30.342859 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.343367 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.344199 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:30.344426 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:30.344494 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:30.347285 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:54479: BlockingRecv error: recv got EOF from 127.0.203.195:54479 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 12:00:31 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:31 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:31.086144 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:31.077537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:35331)
0619 12:00:31.077743 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:31.077765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:31.077829 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:31.078006 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:31.078007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:31.078184 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:31.078235 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:31.078237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.078962 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.078963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:31.079504 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:31.079505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.079904 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.079905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:31.079951 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:31.083686 (+ 3735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:31.083692 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:31.086078 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
W20260619 12:00:31.086386 10031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:31.077590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:35331 (local address 127.0.203.254:38415)
0619 12:00:31.077830 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:31.077832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:31.077870 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:00:31.077900 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:31.077901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:31.077914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:31.077980 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:31.078326 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.078886 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.079608 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.079821 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.079880 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:31.086274 (+ 6394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:35331: BlockingRecv error: recv got EOF from 127.0.203.193:35331 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
Jun 19 12:00:31 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:31 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:31.212654 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:31.206541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:56817)
0619 12:00:31.206819 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:31.206836 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:31.206921 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:31.207179 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:31.207181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:31.207375 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:31.207443 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:31.207445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.208281 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.208283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:31.209024 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:31.209026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.209459 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.209460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:31.209513 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:31.210053 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:31.210058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:31.212549 (+ 2491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":156,"threads_started":1}
W20260619 12:00:31.212741 10031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:31.206736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:56817 (local address 127.0.203.254:38415)
0619 12:00:31.206912 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:31.206915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:31.206934 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:00:31.206975 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:31.206976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:31.206993 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:31.207090 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:31.207558 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.208074 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.209113 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.209368 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.209457 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:31.212670 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:56817: BlockingRecv error: recv got EOF from 127.0.203.194:56817 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Jun 19 12:00:31 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:31 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:31.353312 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:31.347802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:33945)
0619 12:00:31.347958 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:31.347976 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:31.348043 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:31.348185 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:31.348186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:31.348362 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:31.348418 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:31.348420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.349289 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.349290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:31.349883 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:31.349885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.350270 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.350298 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:31.350349 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:31.350885 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:31.350890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:31.353232 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
W20260619 12:00:31.353420 10031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:31.347860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:33945 (local address 127.0.203.254:38415)
0619 12:00:31.347943 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:31.347946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:31.348045 (+ 99us) server_negotiation.cc:407] Connection header received
0619 12:00:31.348097 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:31.348098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:31.348112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:31.348176 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:31.348493 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.349187 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.349977 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:31.350163 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:31.350224 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:31.353350 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:33945: BlockingRecv error: recv got EOF from 127.0.203.195:33945 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 12:00:32 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:32 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:32.092151 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:32.086759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:53339)
0619 12:00:32.086929 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:32.086941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:32.087001 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:32.087240 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:32.087241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:32.087444 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:32.087547 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:32.087549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.088323 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.088323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:32.088949 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:32.088951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.089402 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.089403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:32.089463 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:32.090066 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:32.090072 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:32.092075 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
W20260619 12:00:32.092267 10035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:32.086861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:53339 (local address 127.0.203.254:38415)
0619 12:00:32.087048 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:32.087050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:32.087061 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:32.087086 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:32.087087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:32.087097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:32.087185 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:32.087679 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.088204 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.089064 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.089285 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.089369 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:32.092203 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:53339: BlockingRecv error: recv got EOF from 127.0.203.193:53339 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
Jun 19 12:00:32 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:32 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:32.218816 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:32.213286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:53123)
0619 12:00:32.213650 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:32.213662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:32.213789 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:32.213980 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:32.213981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:32.214197 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:32.214254 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:32.214256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.214967 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.214968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:32.215509 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:32.215510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.215885 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.215886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:32.215938 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:32.216569 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:32.216574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:32.218706 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":125,"threads_started":1}
W20260619 12:00:32.218858 10035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:32.213399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:53123 (local address 127.0.203.254:38415)
0619 12:00:32.213602 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:32.213605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:32.213763 (+ 158us) server_negotiation.cc:407] Connection header received
0619 12:00:32.213879 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:32.213880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:32.213901 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:32.213974 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:32.214349 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.214884 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.215603 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.215801 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.215886 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:32.218788 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:53123: BlockingRecv error: recv got EOF from 127.0.203.194:53123 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Jun 19 12:00:32 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:32 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:32.359468 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:32.353959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:58647)
0619 12:00:32.354165 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:32.354201 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:32.354281 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:32.354469 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:32.354470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:32.354688 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:32.354747 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:32.354749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.355475 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.355477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:32.356018 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:32.356019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.356381 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.356382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:32.356458 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:32.357108 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:32.357113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:32.359401 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":131,"threads_started":1}
W20260619 12:00:32.359660 10035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:32.354149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:58647 (local address 127.0.203.254:38415)
0619 12:00:32.354241 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:32.354244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:32.354263 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:00:32.354378 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:32.354379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:32.354400 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:32.354463 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:32.354843 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.355384 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.356107 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:32.356298 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:32.356383 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:32.359504 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:58647: BlockingRecv error: recv got EOF from 127.0.203.195:58647 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 12:00:33 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:33 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:33.098315 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:33.092918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:58071)
0619 12:00:33.093099 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:33.093117 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:33.093229 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:33.093446 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:33.093447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:33.093663 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:33.093726 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:33.093728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.094465 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.094466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:33.095061 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:33.095063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.095449 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.095450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:33.095502 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:33.096052 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:33.096057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:33.098192 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
W20260619 12:00:33.098351 10039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:33.093032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:58071 (local address 127.0.203.254:38415)
0619 12:00:33.093213 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:33.093216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:33.093227 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:33.093289 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:33.093290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:33.093318 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:33.093393 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:33.093850 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.094362 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.095157 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.095368 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.095429 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:33.098274 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:58071: BlockingRecv error: recv got EOF from 127.0.203.193:58071 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
Jun 19 12:00:33 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:33 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:33.225991 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:33.219580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:57785)
0619 12:00:33.219764 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:33.219778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:33.219877 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:33.220045 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:33.220045 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:33.220281 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:33.220347 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:33.220348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.221334 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.221335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:33.222022 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:33.222024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.222442 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.222443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:33.222498 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:33.223144 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:33.223150 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:33.225903 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
W20260619 12:00:33.226156 10039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:33.219766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:57785 (local address 127.0.203.254:38415)
0619 12:00:33.219843 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:33.219845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:33.219861 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:00:33.219944 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:33.219945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:33.219956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:33.220021 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:33.220431 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.221215 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.222136 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.222348 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.222421 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:33.226046 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:57785: BlockingRecv error: recv got EOF from 127.0.203.194:57785 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 12:00:33 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:33 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:33.366948 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:33.360085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:50297)
0619 12:00:33.360270 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:33.360286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:33.360375 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:33.360561 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:33.360562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:33.360818 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:33.360936 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:33.360939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.361662 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.361663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:33.362321 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:33.362323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.363058 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.363060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:33.363153 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:33.363684 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:33.363689 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:33.366866 (+ 3177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":112,"threads_started":1}
W20260619 12:00:33.367177 10039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:33.360228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:50297 (local address 127.0.203.254:38415)
0619 12:00:33.360290 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:33.360293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:33.360374 (+ 81us) server_negotiation.cc:407] Connection header received
0619 12:00:33.360440 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:33.360441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:33.360458 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:33.360552 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:33.361028 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.361575 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.362432 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:33.362794 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:33.363032 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:33.367109 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:50297: BlockingRecv error: recv got EOF from 127.0.203.195:50297 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 12:00:34 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:34 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:34.104663 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:34.099010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:49393)
0619 12:00:34.099207 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:34.099224 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:34.099306 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:34.099532 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:34.099533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:34.099753 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:34.099822 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:34.099824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.100574 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.100574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:34.101190 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:34.101191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.101647 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.101648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:34.101702 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:34.102485 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:34.102490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:34.104560 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
W20260619 12:00:34.104935 10043 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:34.099102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:49393 (local address 127.0.203.254:38415)
0619 12:00:34.099310 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:34.099313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:34.099327 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:00:34.099358 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:34.099359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:34.099372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:34.099461 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:34.099945 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.100478 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.101317 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.101539 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.101613 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:34.104857 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:49393: BlockingRecv error: recv got EOF from 127.0.203.193:49393 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
Jun 19 12:00:34 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:34 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:34.231952 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:34.226651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:50271)
0619 12:00:34.226823 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:34.226835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:34.226897 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:34.227127 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:34.227128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:34.227333 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:34.227395 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:34.227397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.228210 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.228211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:34.228893 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:34.228895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.229280 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.229281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:34.229329 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:34.229853 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:34.229858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:34.231846 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
W20260619 12:00:34.232141 10043 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:34.226834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:50271 (local address 127.0.203.254:38415)
0619 12:00:34.226902 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:34.226904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:34.226967 (+ 63us) server_negotiation.cc:407] Connection header received
0619 12:00:34.226995 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:34.226996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:34.227007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:34.227066 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:34.227512 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.228108 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.228981 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.229194 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.229270 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:34.231962 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:50271: BlockingRecv error: recv got EOF from 127.0.203.194:50271 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 12:00:34 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:34 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:34.372807 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:34.367476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:44687)
0619 12:00:34.367654 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:34.367677 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:34.367751 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:34.368012 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:34.368013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:34.368179 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:34.368245 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:34.368247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.369062 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.369063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:34.369619 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:34.369621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.369982 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.369983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:34.370034 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:34.370539 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:34.370544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:34.372694 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
W20260619 12:00:34.372839 10043 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:34.367610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:44687 (local address 127.0.203.254:38415)
0619 12:00:34.367774 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:34.367776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:34.367828 (+ 52us) server_negotiation.cc:407] Connection header received
0619 12:00:34.367858 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:34.367859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:34.367876 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:34.367934 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:34.368354 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.368976 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.369724 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:34.369908 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:34.369985 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:34.372774 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:44687: BlockingRecv error: recv got EOF from 127.0.203.195:44687 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 12:00:35 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:35 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:35.110954 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:35.105320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:54471)
0619 12:00:35.105468 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:35.105479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:35.105532 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:35.105783 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:35.105784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:35.105974 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:35.106026 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:35.106027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.106857 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.106858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:35.107436 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:35.107437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.107865 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.107866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:35.107916 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:35.108585 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:35.108590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:35.110829 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260619 12:00:35.111037 10047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:35.105397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:54471 (local address 127.0.203.254:38415)
0619 12:00:35.105574 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:35.105577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:35.105589 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:35.105618 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:35.105619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:35.105632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:35.105729 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:35.106146 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.106720 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.107544 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.107754 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.107818 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:35.110956 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:54471: BlockingRecv error: recv got EOF from 127.0.203.193:54471 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
Jun 19 12:00:35 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:35 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:35.237655 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:35.232620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:46337)
0619 12:00:35.232787 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:35.232804 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:35.232870 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:35.233069 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:35.233070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:35.233293 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:35.233349 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:35.233351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.234087 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.234088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:35.234633 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:35.234634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.235023 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.235025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:35.235077 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:35.235604 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:35.235609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:35.237584 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
W20260619 12:00:35.237854 10047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:35.232756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:46337 (local address 127.0.203.254:38415)
0619 12:00:35.232856 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:35.232859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:35.232871 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:35.232965 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:35.232967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:35.232995 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:35.233063 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:35.233434 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.233994 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.234719 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.234931 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.235026 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:35.237771 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:46337: BlockingRecv error: recv got EOF from 127.0.203.194:46337 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 12:00:35 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:35 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:35.379134 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:35.373606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:59623)
0619 12:00:35.373865 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:35.373884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:35.374028 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:35.374227 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:35.374228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:35.374430 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:35.374477 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:35.374479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.375243 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.375244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:35.375802 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:35.375803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.376203 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.376204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:35.376265 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:35.376875 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:35.376880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:35.379063 (+ 2183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
W20260619 12:00:35.379290 10047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:35.373750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:59623 (local address 127.0.203.254:38415)
0619 12:00:35.373836 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:35.373838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:35.374017 (+ 179us) server_negotiation.cc:407] Connection header received
0619 12:00:35.374106 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:35.374107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:35.374133 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:35.374220 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:35.374569 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.375118 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.375894 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:35.376087 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:35.376183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:35.379227 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:59623: BlockingRecv error: recv got EOF from 127.0.203.195:59623 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 12:00:36 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:36 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:36.120993 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:36.111613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:47389)
0619 12:00:36.111807 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:36.111821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:36.111947 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:36.112296 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:36.112297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:36.112539 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:36.112610 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:36.112613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.113442 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.113443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:36.114049 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:36.114051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.114495 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.114496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:36.114559 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:36.118572 (+ 4013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:36.118578 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:36.120918 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
W20260619 12:00:36.121249 10051 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:36.111742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:47389 (local address 127.0.203.254:38415)
0619 12:00:36.111892 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:36.111895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:36.111917 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:00:36.112140 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:36.112141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:36.112154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:36.112246 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:36.112778 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.113309 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.114161 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.114389 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.114465 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:36.121164 (+ 6699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:47389: BlockingRecv error: recv got EOF from 127.0.203.193:47389 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
Jun 19 12:00:36 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:36 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:36.244284 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:36.238355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:56453)
0619 12:00:36.238587 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:36.238600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:36.238681 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:36.238916 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:36.238918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:36.239128 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:36.239193 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:36.239194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.239971 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.239972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:36.240815 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:36.240820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.241209 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.241210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:36.241265 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:36.241903 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:36.241908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:36.244201 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":150,"threads_started":1}
W20260619 12:00:36.244493 10051 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:36.238464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:56453 (local address 127.0.203.254:38415)
0619 12:00:36.238545 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:36.238546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:36.238709 (+ 163us) server_negotiation.cc:407] Connection header received
0619 12:00:36.238753 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:36.238754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:36.238770 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:36.238860 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:36.239345 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.239868 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.240917 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.241110 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.241177 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:36.244418 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:56453: BlockingRecv error: recv got EOF from 127.0.203.194:56453 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 12:00:36 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:36 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.195@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:36.385483 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:36.379869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.195:54305)
0619 12:00:36.380112 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:36.380127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:36.380235 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:36.380403 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:36.380403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:36.380615 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:36.380738 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:36.380746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.381454 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.381455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:36.382036 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:36.382038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.382424 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.382424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:36.382468 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:36.383065 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:36.383070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:36.385386 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
W20260619 12:00:36.385643 10051 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:36.380017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:54305 (local address 127.0.203.254:38415)
0619 12:00:36.380100 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:36.380103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:36.380239 (+ 136us) server_negotiation.cc:407] Connection header received
0619 12:00:36.380308 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:36.380309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:36.380323 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:36.380372 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:36.380840 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.381364 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.382130 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:36.382345 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:36.382408 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:36.385526 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.195:54305: BlockingRecv error: recv got EOF from 127.0.203.195:54305 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 12:00:37 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:37 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.193@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:37.127760 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.121804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.193:42633)
0619 12:00:37.122042 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:37.122058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:37.122186 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:37.122437 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:37.122439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:37.122615 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:37.122666 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.122668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.123425 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.123426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.124035 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.124037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.124517 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.124518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.124579 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.125390 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:37.125395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:37.127684 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
W20260619 12:00:37.128027 10055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:37.121901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:42633 (local address 127.0.203.254:38415)
0619 12:00:37.122021 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:37.122028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:37.122191 (+ 163us) server_negotiation.cc:407] Connection header received
0619 12:00:37.122250 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:37.122251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:37.122264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:37.122367 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:37.122758 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.123314 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.124163 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.124388 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.124471 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.127929 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.193:42633: BlockingRecv error: recv got EOF from 127.0.203.193:42633 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":48,"threads_started":1}
Jun 19 12:00:37 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:00:37 dist-test-slave-fbq8 krb5kdc[8995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.203.194@KRBTEST.COM for oryx/127.0.203.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:00:37.250504 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.245082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:38415 (local address 127.0.203.194:38389)
0619 12:00:37.245264 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:37.245278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:37.245367 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:37.245531 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:37.245532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:37.245769 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:37.245818 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.245819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.246556 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.246557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.247101 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.247102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.247482 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.247483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.247538 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.248139 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:37.248143 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:37.250403 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.203.254:38415: Server oryx/127.0.203.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
W20260619 12:00:37.250574 10055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:37.245274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:38389 (local address 127.0.203.254:38415)
0619 12:00:37.245380 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:37.245382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:37.245394 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:00:37.245438 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:37.245439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:37.245458 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:37.245525 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:37.245906 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.246459 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.247192 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.247394 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.247480 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.250509 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.203.194:38389: BlockingRecv error: recv got EOF from 127.0.203.194:38389 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 12:00:37.289570 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9578
I20260619 12:00:37.294906 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9730
I20260619 12:00:37.300554 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9864
I20260619 12:00:37.305739 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 9508
2026-06-19T12:00:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17688 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T12:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:37Z Disabled control of system clock
I20260619 12:00:37.337785 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:37855
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:37855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 12:00:37.427052 10060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:37.427271 10060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:37.427292 10060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:37.428596 10060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:37.428660 10060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:37.428674 10060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:37.428689 10060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:37.428700 10060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:37.430116 10060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:37855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:37.430392 10060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:37.430742 10060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:37.437240 10068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:37.437288 10065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:37.437306 10060 server_base.cc:1034] running on GCE node
W20260619 12:00:37.437263 10066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:37.437953 10060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:37.438323 10060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:37.439538 10060 hybrid_clock.cc:648] HybridClock initialized: now 1781870437439527 us; error 64 us; skew 500 ppm
I20260619 12:00:37.444334 10060 webserver.cc:460] Webserver started at http://127.0.203.254:42535/ using document root <none> and password file <none>
I20260619 12:00:37.444798 10060 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:37.444881 10060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:37.445039 10060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:37.446189 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "f3f35ff891bd427696b3d842e5ec6123"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "934224612baf5d0dbc3830e862a10d3f"
server_key_iv: "e61dfa3cab243cbb525eeada81e2600a"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.446569 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "f3f35ff891bd427696b3d842e5ec6123"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "934224612baf5d0dbc3830e862a10d3f"
server_key_iv: "e61dfa3cab243cbb525eeada81e2600a"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.448002 10060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:37.448987 10074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.449203 10060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:37.449313 10060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "f3f35ff891bd427696b3d842e5ec6123"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "934224612baf5d0dbc3830e862a10d3f"
server_key_iv: "e61dfa3cab243cbb525eeada81e2600a"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.449405 10060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:37.456022 10060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:37.456331 10060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:37.456496 10060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:37.461104 10060 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:37855
I20260619 12:00:37.461159 10126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:37855 every 8 connection(s)
I20260619 12:00:37.461565 10060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:37.462385 10127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:37.463354 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10060
I20260619 12:00:37.463474 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:37.463703 815 external_mini_cluster.cc:1428] Setting key b9680e4b0185772796121ac2488b2715
I20260619 12:00:37.467399 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123: Bootstrap starting.
I20260619 12:00:37.468477 10127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:37.468888 10127 log.cc:824] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:37.469918 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123: No bootstrap required, opened a new log
I20260619 12:00:37.472849 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.464357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.203.254:37855)
0619 12:00:37.464614 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:37.464624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:37.464951 (+ 327us) server_negotiation.cc:407] Connection header received
0619 12:00:37.466047 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:37.466056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:37.466148 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:37.466317 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:37.466758 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.468347 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.469484 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.470423 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.470577 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.471864 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:37.471869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:37.471869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:37.472123 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:37.472177 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:37.472193 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:37.472283 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:37.472422 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 12:00:37.472537 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260619 12:00:37.472932 10127 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f35ff891bd427696b3d842e5ec6123" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 37855 } }
I20260619 12:00:37.473101 10127 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:37.473134 10127 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f35ff891bd427696b3d842e5ec6123, State: Initialized, Role: FOLLOWER
I20260619 12:00:37.473256 10127 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f35ff891bd427696b3d842e5ec6123" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 37855 } }
I20260619 12:00:37.473342 10127 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:37.473377 10127 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:37.473429 10127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:37.474385 10127 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f35ff891bd427696b3d842e5ec6123" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 37855 } }
I20260619 12:00:37.474507 10127 leader_election.cc:302] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f35ff891bd427696b3d842e5ec6123; no voters:
I20260619 12:00:37.474694 10127 leader_election.cc:288] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:37.474758 10132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:37.474944 10132 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [term 1 LEADER]: Becoming Leader. State: Replica: f3f35ff891bd427696b3d842e5ec6123, State: Running, Role: LEADER
I20260619 12:00:37.475026 10127 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:37.475047 10132 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f35ff891bd427696b3d842e5ec6123" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 37855 } }
I20260619 12:00:37.475772 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f35ff891bd427696b3d842e5ec6123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f35ff891bd427696b3d842e5ec6123" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 37855 } } }
I20260619 12:00:37.475885 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:37.475776 10134 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f35ff891bd427696b3d842e5ec6123. Latest consensus state: current_term: 1 leader_uuid: "f3f35ff891bd427696b3d842e5ec6123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f35ff891bd427696b3d842e5ec6123" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 37855 } } }
I20260619 12:00:37.476140 10141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:37.476167 10134 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:37.477061 10141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:37.479074 10141 catalog_manager.cc:1347] Generated new cluster ID: d9e7d22a351c463fa7bf97162046e082
I20260619 12:00:37.479139 10141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:37.501529 10141 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:37.502218 10141 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:37.511076 10141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f35ff891bd427696b3d842e5ec6123: Generated new TSK 0
I20260619 12:00:37.511277 10141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:00:37.517158 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:37855
--builtin_ntp_servers=127.0.203.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 12:00:37.614108 10151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:37.614346 10151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:37.614368 10151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:37.615635 10151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:37.615679 10151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:37.615705 10151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:37.617197 10151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:37.617478 10151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:37.617835 10151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:37.623456 10159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:37.623448 10156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:37.623505 10157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:37.623696 10151 server_base.cc:1034] running on GCE node
I20260619 12:00:37.624251 10151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:37.624527 10151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:37.625732 10151 hybrid_clock.cc:648] HybridClock initialized: now 1781870437625703 us; error 49 us; skew 500 ppm
I20260619 12:00:37.627291 10151 webserver.cc:460] Webserver started at http://127.0.203.193:35491/ using document root <none> and password file <none>
I20260619 12:00:37.627574 10151 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:37.627648 10151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:37.627806 10151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:37.629070 10151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "d5c489ea0da0429f85451d338b620e41"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "44be197431174342e8eba011c3fb19d2"
server_key_iv: "87e867e7d9f6c74b0631cfa5e63438f5"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.629462 10151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "d5c489ea0da0429f85451d338b620e41"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "44be197431174342e8eba011c3fb19d2"
server_key_iv: "87e867e7d9f6c74b0631cfa5e63438f5"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.631143 10151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:37.632085 10165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.632323 10151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:37.632441 10151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "d5c489ea0da0429f85451d338b620e41"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "44be197431174342e8eba011c3fb19d2"
server_key_iv: "87e867e7d9f6c74b0631cfa5e63438f5"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.632555 10151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:37.649349 10151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:37.649660 10151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:37.649777 10151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:37.650029 10151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:37.650422 10151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:37.650484 10151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.650519 10151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:37.650548 10151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.657132 10151 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:40675
I20260619 12:00:37.657270 10278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:40675 every 8 connection(s)
I20260619 12:00:37.657610 10151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:37.664062 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10151
I20260619 12:00:37.664175 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:37.664310 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.658658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:54989 (local address 127.0.203.254:37855)
0619 12:00:37.658814 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:37.658818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:37.659049 (+ 231us) server_negotiation.cc:407] Connection header received
0619 12:00:37.659725 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:37.659727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:37.659742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:37.659802 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:37.660576 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.661373 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.662611 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.662850 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.662936 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.663945 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:37.663946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:37.663947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:37.663953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:37.663968 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:37.663972 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:37.664047 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:37.664181 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 12:00:37.664222 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 12:00:37.664402 815 external_mini_cluster.cc:1428] Setting key 6e94335e1b3d6968c2c18a3be9d133f8
I20260619 12:00:37.664554 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.658567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:37855 (local address 127.0.203.193:54989)
0619 12:00:37.658884 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:37.658927 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:37.659586 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:37.659898 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:37.659902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:37.659912 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:37.660469 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.660473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.661561 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.661563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.662495 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.662498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.662943 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.662944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.663102 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.663740 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:37.663741 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:37.663778 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:37.663804 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:37.664064 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:37.664070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:37.664073 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:37.664175 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 12:00:37.664303 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260619 12:00:37.665190 10279 heartbeater.cc:347] Connected to a master server at 127.0.203.254:37855
I20260619 12:00:37.665328 10279 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:37.665521 10279 heartbeater.cc:511] Master 127.0.203.254:37855 requested a full tablet report, sending...
I20260619 12:00:37.665515 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:37855
--builtin_ntp_servers=127.0.203.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 12:00:37.666113 10091 ts_manager.cc:194] Registered new tserver with Master: d5c489ea0da0429f85451d338b620e41 (127.0.203.193:40675)
I20260619 12:00:37.666843 10091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.193:54989
W20260619 12:00:37.754205 10282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:37.754463 10282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:37.754503 10282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:37.755690 10282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:37.755769 10282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:37.755817 10282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:37.757314 10282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:37.757599 10282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:37.757944 10282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:37.764155 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:37.764243 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:37.764163 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:37.764775 10282 server_base.cc:1034] running on GCE node
I20260619 12:00:37.765040 10282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:37.765357 10282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:37.766585 10282 hybrid_clock.cc:648] HybridClock initialized: now 1781870437766571 us; error 48 us; skew 500 ppm
I20260619 12:00:37.768316 10282 webserver.cc:460] Webserver started at http://127.0.203.194:38355/ using document root <none> and password file <none>
I20260619 12:00:37.768610 10282 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:37.768716 10282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:37.768882 10282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:37.770160 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "02e2b61162394b9689c888a5ae0268a3"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "7470ea41fb2d657ad2962b563abaecc9"
server_key_iv: "ecd584d393c75d376be8f1b1effbc89d"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.770579 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "02e2b61162394b9689c888a5ae0268a3"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "7470ea41fb2d657ad2962b563abaecc9"
server_key_iv: "ecd584d393c75d376be8f1b1effbc89d"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.772223 10282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:37.773135 10296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.773373 10282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:00:37.773484 10282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "02e2b61162394b9689c888a5ae0268a3"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "7470ea41fb2d657ad2962b563abaecc9"
server_key_iv: "ecd584d393c75d376be8f1b1effbc89d"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.773563 10282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:37.800228 10282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:37.800681 10282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:37.800864 10282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:37.801211 10282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:37.801641 10282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:37.801677 10282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.801702 10282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:37.801755 10282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.808938 10282 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:45825
I20260619 12:00:37.809059 10409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:45825 every 8 connection(s)
I20260619 12:00:37.809404 10282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:37.812366 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10282
I20260619 12:00:37.812446 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:37.812713 815 external_mini_cluster.cc:1428] Setting key 5e5ac06bd1074f50f8bc017c1090c6e3
I20260619 12:00:37.813908 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:37855
--builtin_ntp_servers=127.0.203.212:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 12:00:37.815894 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.810313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:40901 (local address 127.0.203.254:37855)
0619 12:00:37.810402 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:37.810404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:37.810686 (+ 282us) server_negotiation.cc:407] Connection header received
0619 12:00:37.811256 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:37.811257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:37.811268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:37.811364 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:37.812249 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.812971 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.814269 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.814485 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.814564 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.815596 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:37.815597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:37.815598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:37.815604 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:37.815618 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:37.815622 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:37.815676 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:37.815773 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 12:00:37.815808 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:00:37.816166 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.810302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:37855 (local address 127.0.203.194:40901)
0619 12:00:37.810534 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:37.810582 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:37.811162 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:37.811495 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:37.811500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:37.811521 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:37.812084 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.812088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.813100 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.813102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.814164 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.814168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.814573 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.814574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.814716 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.815445 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:37.815445 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:37.815477 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:37.815504 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:37.815684 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:37.815689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:37.815692 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:37.815776 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:00:37.815932 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
I20260619 12:00:37.817046 10410 heartbeater.cc:347] Connected to a master server at 127.0.203.254:37855
I20260619 12:00:37.817176 10410 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:37.817400 10410 heartbeater.cc:511] Master 127.0.203.254:37855 requested a full tablet report, sending...
I20260619 12:00:37.817866 10091 ts_manager.cc:194] Registered new tserver with Master: 02e2b61162394b9689c888a5ae0268a3 (127.0.203.194:45825)
I20260619 12:00:37.818274 10091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.194:40901
W20260619 12:00:37.906206 10413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:37.906450 10413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:37.906471 10413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:37.907668 10413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:37.907718 10413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:37.907744 10413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:37.909241 10413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:37.909518 10413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:37.909921 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:37.915663 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:37.915762 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:37.915737 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:37.915884 10413 server_base.cc:1034] running on GCE node
I20260619 12:00:37.916332 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:37.916616 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:37.917837 10413 hybrid_clock.cc:648] HybridClock initialized: now 1781870437917810 us; error 48 us; skew 500 ppm
I20260619 12:00:37.919543 10413 webserver.cc:460] Webserver started at http://127.0.203.195:43007/ using document root <none> and password file <none>
I20260619 12:00:37.919873 10413 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:37.919962 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:37.920145 10413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:37.921456 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "9581ab224a7e42c0a4ac4e9ccdd8c8c6"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "d3ad3cd7caeb2650e34c46d6c71ce2b8"
server_key_iv: "dbc6c601f922def7ddbaeac4bdb6cccc"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.921854 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "9581ab224a7e42c0a4ac4e9ccdd8c8c6"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "d3ad3cd7caeb2650e34c46d6c71ce2b8"
server_key_iv: "dbc6c601f922def7ddbaeac4bdb6cccc"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.923375 10413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:00:37.924222 10427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.924477 10413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:37.924587 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "9581ab224a7e42c0a4ac4e9ccdd8c8c6"
format_stamp: "Formatted at 2026-06-19 12:00:37 on dist-test-slave-fbq8"
server_key: "d3ad3cd7caeb2650e34c46d6c71ce2b8"
server_key_iv: "dbc6c601f922def7ddbaeac4bdb6cccc"
server_key_version: "encryptionkey@0"
I20260619 12:00:37.924705 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:37.939862 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:37.940285 10413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:37.940447 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:37.940809 10413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:37.941205 10413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:37.941265 10413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.941310 10413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:37.941336 10413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:37.949349 10413 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:33491
I20260619 12:00:37.949458 10540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:33491 every 8 connection(s)
I20260619 12:00:37.949820 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:37.950412 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10413
I20260619 12:00:37.950517 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:37.950784 815 external_mini_cluster.cc:1428] Setting key f98716fde0c10c7ac9666cfced36c892
I20260619 12:00:37.955809 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.950722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:50955 (local address 127.0.203.254:37855)
0619 12:00:37.950770 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:37.950772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:37.951139 (+ 367us) server_negotiation.cc:407] Connection header received
0619 12:00:37.951924 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:37.951925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:37.951937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:37.951980 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:37.952878 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.953405 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.954416 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.954595 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.954650 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.955496 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:37.955497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:37.955497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:37.955502 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:37.955519 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:37.955522 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:37.955568 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:37.955720 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 12:00:37.955743 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 12:00:37.956121 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:37.950742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:37855 (local address 127.0.203.195:50955)
0619 12:00:37.950994 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:37.951050 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:37.951802 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:37.952063 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:37.952068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:37.952080 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:37.952790 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.952795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.953491 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.953492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.954336 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:37.954338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:37.954672 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:37.954673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:37.954783 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:37.955348 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:37.955349 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:37.955380 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:37.955404 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:37.955622 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:37.955627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:37.955630 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:37.955710 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:00:37.955815 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260619 12:00:37.956763 10541 heartbeater.cc:347] Connected to a master server at 127.0.203.254:37855
I20260619 12:00:37.956905 10541 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:37.957131 10541 heartbeater.cc:511] Master 127.0.203.254:37855 requested a full tablet report, sending...
I20260619 12:00:37.957571 10091 ts_manager.cc:194] Registered new tserver with Master: 9581ab224a7e42c0a4ac4e9ccdd8c8c6 (127.0.203.195:33491)
I20260619 12:00:37.957945 10091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.195:50955
I20260619 12:00:37.962697 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:00:37.965034 10130 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:37.964226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.0.203.254:37855)
0619 12:00:37.964294 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:37.964296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:37.964492 (+ 196us) server_negotiation.cc:407] Connection header received
0619 12:00:37.964521 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:37.964522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:37.964533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:37.964595 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:37.964920 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41386: BlockingRecv error: recv got EOF from 127.0.0.1:41386 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 12:00:37.965116 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10151
I20260619 12:00:37.970831 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10282
I20260619 12:00:37.976863 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10413
I20260619 12:00:37.986213 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10060
2026-06-19T12:00:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (680 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T12:00:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:38Z Disabled control of system clock
I20260619 12:00:38.016892 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:35835
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:00:38.102010 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:38.102241 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:38.102265 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:38.103509 10552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:38.103569 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:38.103581 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:38.103595 10552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:38.103609 10552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:38.105149 10552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:35835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:38.105433 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:38.105957 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:38.111999 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.111980 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.111980 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:38.113103 10552 server_base.cc:1034] running on GCE node
I20260619 12:00:38.113418 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:38.113721 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:38.114924 10552 hybrid_clock.cc:648] HybridClock initialized: now 1781870438114914 us; error 42 us; skew 500 ppm
I20260619 12:00:38.116372 10552 webserver.cc:460] Webserver started at http://127.0.203.254:33019/ using document root <none> and password file <none>
I20260619 12:00:38.116729 10552 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:38.116801 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:38.116988 10552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:38.118106 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "93735309a05c4908bc4c6a498d591420"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "ab161a6b387a5d558d405b89552db023"
server_key_iv: "05614b1fd3ff15589b963a68b450d6e0"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.118476 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "93735309a05c4908bc4c6a498d591420"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "ab161a6b387a5d558d405b89552db023"
server_key_iv: "05614b1fd3ff15589b963a68b450d6e0"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.119941 10552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:00:38.120887 10566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.121142 10552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:38.121249 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "93735309a05c4908bc4c6a498d591420"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "ab161a6b387a5d558d405b89552db023"
server_key_iv: "05614b1fd3ff15589b963a68b450d6e0"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.121335 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:38.124816 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:38.125106 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:38.125253 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:38.129698 10552 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:35835
I20260619 12:00:38.129801 10618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:35835 every 8 connection(s)
I20260619 12:00:38.130136 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:38.130848 10619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:38.132007 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10552
I20260619 12:00:38.132141 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:38.132360 815 external_mini_cluster.cc:1428] Setting key 813c30411250777fa76a71a37f079a09
I20260619 12:00:38.134344 10619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420: Bootstrap starting.
I20260619 12:00:38.135087 10619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:38.135443 10619 log.cc:824] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:38.135782 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:38.133088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.0.203.254:35835)
0619 12:00:38.133374 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:38.133385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:38.133423 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:00:38.134089 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:38.134097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:38.134150 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:38.134318 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:38.135085 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:38.135088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:38.135089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:38.135094 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:38.135140 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:38.135154 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:38.135255 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:38.135408 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 12:00:38.135507 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":133,"threads_started":1}
I20260619 12:00:38.136461 10619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420: No bootstrap required, opened a new log
I20260619 12:00:38.138154 10619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93735309a05c4908bc4c6a498d591420" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35835 } }
I20260619 12:00:38.138334 10619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:38.138357 10619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93735309a05c4908bc4c6a498d591420, State: Initialized, Role: FOLLOWER
I20260619 12:00:38.138516 10619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93735309a05c4908bc4c6a498d591420" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35835 } }
I20260619 12:00:38.138615 10619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:38.138666 10619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:38.138720 10619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:38.139705 10619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93735309a05c4908bc4c6a498d591420" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35835 } }
I20260619 12:00:38.139829 10619 leader_election.cc:302] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93735309a05c4908bc4c6a498d591420; no voters:
I20260619 12:00:38.140022 10619 leader_election.cc:288] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:38.140298 10624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:38.140358 10619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:38.140522 10624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [term 1 LEADER]: Becoming Leader. State: Replica: 93735309a05c4908bc4c6a498d591420, State: Running, Role: LEADER
I20260619 12:00:38.140666 10624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93735309a05c4908bc4c6a498d591420" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35835 } }
I20260619 12:00:38.141366 10626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93735309a05c4908bc4c6a498d591420. Latest consensus state: current_term: 1 leader_uuid: "93735309a05c4908bc4c6a498d591420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93735309a05c4908bc4c6a498d591420" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35835 } } }
I20260619 12:00:38.141477 10626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:38.141485 10625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93735309a05c4908bc4c6a498d591420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93735309a05c4908bc4c6a498d591420" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35835 } } }
I20260619 12:00:38.141611 10625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420 [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:38.141964 10635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:38.142503 10635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:38.144289 10635 catalog_manager.cc:1347] Generated new cluster ID: 1fccbd5459a8408fae6a525d66ee87e2
I20260619 12:00:38.144353 10635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:00:38.150966 10635 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:38.151173 10635 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:38.154273 10635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93735309a05c4908bc4c6a498d591420: Generated new TSK 0
I20260619 12:00:38.154614 10635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:00:38.158209 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35835
--builtin_ntp_servers=127.0.203.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:00:38.242444 10643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:38.242699 10643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:38.242740 10643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:38.244011 10643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:38.244079 10643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:38.244127 10643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:38.245635 10643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:38.245921 10643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:38.246230 10643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:38.251408 10648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.251544 10649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.251413 10651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:38.251628 10643 server_base.cc:1034] running on GCE node
I20260619 12:00:38.251940 10643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:38.252231 10643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:38.253402 10643 hybrid_clock.cc:648] HybridClock initialized: now 1781870438253391 us; error 27 us; skew 500 ppm
I20260619 12:00:38.254745 10643 webserver.cc:460] Webserver started at http://127.0.203.193:33029/ using document root <none> and password file <none>
I20260619 12:00:38.255007 10643 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:38.255079 10643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:38.255229 10643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:38.256387 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "baf0163c66ed4ebf8487d1326a79709a"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "9198b59c2351665855f24e7693a130f0"
server_key_iv: "415f2b789828f36fdd03ee5e265d7b58"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.256816 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "baf0163c66ed4ebf8487d1326a79709a"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "9198b59c2351665855f24e7693a130f0"
server_key_iv: "415f2b789828f36fdd03ee5e265d7b58"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.258172 10643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:38.258955 10657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.259202 10643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:38.259312 10643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "baf0163c66ed4ebf8487d1326a79709a"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "9198b59c2351665855f24e7693a130f0"
server_key_iv: "415f2b789828f36fdd03ee5e265d7b58"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.259390 10643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:38.264441 10643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:38.264793 10643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:38.264940 10643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:38.265170 10643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:38.265595 10643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:38.265645 10643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.265676 10643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:38.265693 10643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.272701 10643 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:40679
I20260619 12:00:38.272746 10770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:40679 every 8 connection(s)
I20260619 12:00:38.273192 10643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:38.274286 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10643
I20260619 12:00:38.274377 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:38.274605 815 external_mini_cluster.cc:1428] Setting key bbb29fb6097b4c727fd8645cb98b1ada
I20260619 12:00:38.275749 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35835
--builtin_ntp_servers=127.0.203.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 12:00:38.277089 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:38.274188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:46643 (local address 127.0.203.254:35835)
0619 12:00:38.274247 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:38.274250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:38.274529 (+ 279us) server_negotiation.cc:407] Connection header received
0619 12:00:38.275185 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:38.275186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:38.275191 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:38.275237 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:38.276790 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:38.276792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:38.276792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:38.276797 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:38.276812 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:38.276816 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:38.276847 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:38.276970 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 12:00:38.276985 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:00:38.277580 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:38.274139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35835 (local address 127.0.203.193:46643)
0619 12:00:38.274370 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:38.274428 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:38.275079 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:38.275311 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:38.275315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:38.275325 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:38.276577 (+ 1252us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:38.276578 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:38.276621 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:38.276689 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:38.276887 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:38.276892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:38.276895 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:38.277232 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 12:00:38.277308 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260619 12:00:38.278138 10771 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35835
I20260619 12:00:38.278259 10771 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:38.278414 10771 heartbeater.cc:511] Master 127.0.203.254:35835 requested a full tablet report, sending...
I20260619 12:00:38.278950 10583 ts_manager.cc:194] Registered new tserver with Master: baf0163c66ed4ebf8487d1326a79709a (127.0.203.193:40679)
W20260619 12:00:38.358233 10774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:38.358497 10774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:38.358534 10774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:38.359759 10774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:38.359829 10774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:38.359872 10774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:38.361335 10774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:38.361624 10774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:38.361932 10774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:38.367749 10782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.367807 10779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.367877 10780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:38.367807 10774 server_base.cc:1034] running on GCE node
I20260619 12:00:38.368304 10774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:38.368605 10774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:38.369810 10774 hybrid_clock.cc:648] HybridClock initialized: now 1781870438369774 us; error 50 us; skew 500 ppm
I20260619 12:00:38.371461 10774 webserver.cc:460] Webserver started at http://127.0.203.194:44205/ using document root <none> and password file <none>
I20260619 12:00:38.371747 10774 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:38.371819 10774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:38.371973 10774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:38.373198 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "6cce5cd4ca7042ca8cd9d71afadfe9f7"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "f6d20de0424e12112a0123b04df4ffbc"
server_key_iv: "d4dc977a442b1181074353ede77b169c"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.373566 10774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "6cce5cd4ca7042ca8cd9d71afadfe9f7"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "f6d20de0424e12112a0123b04df4ffbc"
server_key_iv: "d4dc977a442b1181074353ede77b169c"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.374944 10774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 12:00:38.375826 10788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.376034 10774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:38.376147 10774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "6cce5cd4ca7042ca8cd9d71afadfe9f7"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "f6d20de0424e12112a0123b04df4ffbc"
server_key_iv: "d4dc977a442b1181074353ede77b169c"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.376226 10774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:38.379904 10774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:38.380213 10774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:38.380366 10774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:38.380595 10774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:38.380982 10774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:38.381038 10774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.381080 10774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:38.381107 10774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.386714 10774 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:35455
I20260619 12:00:38.386795 10901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:35455 every 8 connection(s)
I20260619 12:00:38.387188 10774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:00:38.390487 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:38.388015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:41439 (local address 127.0.203.254:35835)
0619 12:00:38.388058 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:38.388060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:38.388303 (+ 243us) server_negotiation.cc:407] Connection header received
0619 12:00:38.388941 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:38.388942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:38.388948 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:38.388995 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:38.390233 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:38.390234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:38.390235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:38.390239 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:38.390254 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:38.390259 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:38.390307 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:38.390397 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 12:00:38.390411 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 12:00:38.390712 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:38.387987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35835 (local address 127.0.203.194:41439)
0619 12:00:38.388181 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:38.388215 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:38.388795 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:38.389070 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:38.389073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:38.389081 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:38.390059 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:38.390059 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:38.390089 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:38.390127 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:38.390303 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:38.390308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:38.390311 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:00:38.390409 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 12:00:38.390470 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260619 12:00:38.391242 10902 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35835
I20260619 12:00:38.391323 10902 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:38.391492 10902 heartbeater.cc:511] Master 127.0.203.254:35835 requested a full tablet report, sending...
I20260619 12:00:38.391875 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10774
I20260619 12:00:38.391963 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:38.391906 10583 ts_manager.cc:194] Registered new tserver with Master: 6cce5cd4ca7042ca8cd9d71afadfe9f7 (127.0.203.194:35455)
I20260619 12:00:38.392161 815 external_mini_cluster.cc:1428] Setting key dcf827ca6864383b002b099a67ded596
I20260619 12:00:38.393034 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35835
--builtin_ntp_servers=127.0.203.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:00:38.475571 10905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:38.475809 10905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:38.475835 10905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:38.477063 10905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:38.477104 10905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:38.477131 10905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:38.478463 10905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:38.478674 10905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:38.479005 10905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:38.484561 10911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.484550 10910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:38.484717 10913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:38.484730 10905 server_base.cc:1034] running on GCE node
I20260619 12:00:38.485118 10905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:38.485380 10905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:38.486536 10905 hybrid_clock.cc:648] HybridClock initialized: now 1781870438486522 us; error 46 us; skew 500 ppm
I20260619 12:00:38.487903 10905 webserver.cc:460] Webserver started at http://127.0.203.195:42005/ using document root <none> and password file <none>
I20260619 12:00:38.488157 10905 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:38.488202 10905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:38.488375 10905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:38.489535 10905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "6f9103598cd74d8380ed084b87640ca5"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "618936c3ec1067332de30800cf6910f0"
server_key_iv: "2246ae3781318455c1969520e12c53cb"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.489898 10905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "6f9103598cd74d8380ed084b87640ca5"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "618936c3ec1067332de30800cf6910f0"
server_key_iv: "2246ae3781318455c1969520e12c53cb"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.491196 10905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:00:38.491976 10919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.492169 10905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:38.492254 10905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "6f9103598cd74d8380ed084b87640ca5"
format_stamp: "Formatted at 2026-06-19 12:00:38 on dist-test-slave-fbq8"
server_key: "618936c3ec1067332de30800cf6910f0"
server_key_iv: "2246ae3781318455c1969520e12c53cb"
server_key_version: "encryptionkey@0"
I20260619 12:00:38.492308 10905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:38.495667 10905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:38.495890 10905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:38.495986 10905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:38.496133 10905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:38.496382 10905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:38.496416 10905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.496450 10905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:38.496464 10905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:38.502173 10905 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:34319
I20260619 12:00:38.502247 11032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:34319 every 8 connection(s)
I20260619 12:00:38.502570 10905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:00:38.505627 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:38.503323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:38839 (local address 127.0.203.254:35835)
0619 12:00:38.503380 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:38.503382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:38.503602 (+ 220us) server_negotiation.cc:407] Connection header received
0619 12:00:38.504096 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:38.504097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:38.504103 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:38.504164 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:38.505411 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:38.505413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:38.505413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:00:38.505417 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:38.505430 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:00:38.505434 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:38.505497 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:38.505560 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 12:00:38.505573 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:00:38.505831 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:38.503307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35835 (local address 127.0.203.195:38839)
0619 12:00:38.503483 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:38.503516 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:38.503993 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:38.504220 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:38.504223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:38.504231 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:38.505261 (+ 1030us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:00:38.505262 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:38.505289 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:00:38.505318 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:38.505498 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:38.505500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:38.505502 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 12:00:38.505553 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 12:00:38.505600 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260619 12:00:38.506465 11033 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35835
I20260619 12:00:38.506587 11033 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:38.506759 11033 heartbeater.cc:511] Master 127.0.203.254:35835 requested a full tablet report, sending...
I20260619 12:00:38.507166 10583 ts_manager.cc:194] Registered new tserver with Master: 6f9103598cd74d8380ed084b87640ca5 (127.0.203.195:34319)
I20260619 12:00:38.507871 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 10905
I20260619 12:00:38.507995 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:38.508189 815 external_mini_cluster.cc:1428] Setting key 4ba31ce9c63a4d1907c9222ae5433ada
I20260619 12:00:38.509143 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:00:38.511154 10622 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:00:38.510603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.0.203.254:35835)
0619 12:00:38.510654 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:38.510656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:38.510770 (+ 114us) server_negotiation.cc:407] Connection header received
0619 12:00:38.510809 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:38.510810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:38.510814 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:38.510904 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:38.511078 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45416: BlockingRecv error: recv got EOF from 127.0.0.1:45416 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 12:00:38.511423 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10643
I20260619 12:00:38.517066 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10774
I20260619 12:00:38.521747 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10905
I20260619 12:00:38.526718 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 10552
2026-06-19T12:00:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (538 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:38 dist-test-slave-fbq8 krb5kdc[11043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:38 dist-test-slave-fbq8 krb5kdc[11043](info): set up 2 sockets
Jun 19 12:00:38 dist-test-slave-fbq8 krb5kdc[11043](info): commencing operation
krb5kdc: starting...
W20260619 12:00:39.586566 815 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:39 dist-test-slave-fbq8 krb5kdc[11043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:00:39.725840 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:35901
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:35901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:39.810811 11056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:39.811084 11056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:39.811120 11056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:39.812252 11056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:00:39.812325 11056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:39.812350 11056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:39.812368 11056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:00:39.812397 11056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:00:39.813915 11056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:35901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:35901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:39.814190 11056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:39.814420 11056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:39.820062 11062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:39.820099 11064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:39.820102 11056 server_base.cc:1034] running on GCE node
W20260619 12:00:39.820062 11061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:39.820837 11056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:39.821159 11056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:39.822312 11056 hybrid_clock.cc:648] HybridClock initialized: now 1781870439822288 us; error 54 us; skew 500 ppm
Jun 19 12:00:39 dist-test-slave-fbq8 krb5kdc[11043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:39.825239 11056 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:00:39.825773 11056 webserver.cc:460] Webserver started at http://127.0.203.254:37831/ using document root <none> and password file <none>
I20260619 12:00:39.826071 11056 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:39.826171 11056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:39.826334 11056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:39.827561 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "696e1521635149eabba43ae4f9c7d7be"
format_stamp: "Formatted at 2026-06-19 12:00:39 on dist-test-slave-fbq8"
server_key: "eb58843384189f02f93016dac0683e35"
server_key_iv: "69bc351f029056daa6e6fe663426cae6"
server_key_version: "encryptionkey@0"
I20260619 12:00:39.827977 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "696e1521635149eabba43ae4f9c7d7be"
format_stamp: "Formatted at 2026-06-19 12:00:39 on dist-test-slave-fbq8"
server_key: "eb58843384189f02f93016dac0683e35"
server_key_iv: "69bc351f029056daa6e6fe663426cae6"
server_key_version: "encryptionkey@0"
I20260619 12:00:39.829659 11056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 12:00:39.830596 11071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:39.830868 11056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:00:39.830991 11056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "696e1521635149eabba43ae4f9c7d7be"
format_stamp: "Formatted at 2026-06-19 12:00:39 on dist-test-slave-fbq8"
server_key: "eb58843384189f02f93016dac0683e35"
server_key_iv: "69bc351f029056daa6e6fe663426cae6"
server_key_version: "encryptionkey@0"
I20260619 12:00:39.831076 11056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:39.844394 11056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:39.846742 11056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:39.846930 11056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:39.852003 11056 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:35901
I20260619 12:00:39.852048 11123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:35901 every 8 connection(s)
I20260619 12:00:39.852442 11056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:00:39.853593 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:39.856103 11124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be: Bootstrap starting.
I20260619 12:00:39.856784 11124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:39.857101 11124 log.cc:824] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:39.857949 11124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be: No bootstrap required, opened a new log
I20260619 12:00:39.859591 11124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696e1521635149eabba43ae4f9c7d7be" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35901 } }
I20260619 12:00:39.859745 11124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:39.859769 11124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696e1521635149eabba43ae4f9c7d7be, State: Initialized, Role: FOLLOWER
I20260619 12:00:39.859942 11124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696e1521635149eabba43ae4f9c7d7be" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35901 } }
I20260619 12:00:39.860029 11124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:00:39.860054 11124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:00:39.860106 11124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:39.861155 11124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696e1521635149eabba43ae4f9c7d7be" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35901 } }
I20260619 12:00:39.861253 11124 leader_election.cc:302] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 696e1521635149eabba43ae4f9c7d7be; no voters:
I20260619 12:00:39.861498 11124 leader_election.cc:288] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:00:39.861543 11127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:39.861766 11127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [term 1 LEADER]: Becoming Leader. State: Replica: 696e1521635149eabba43ae4f9c7d7be, State: Running, Role: LEADER
I20260619 12:00:39.861792 11124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:00:39.862030 11127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696e1521635149eabba43ae4f9c7d7be" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35901 } }
I20260619 12:00:39.862277 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 11056
I20260619 12:00:39.862380 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:00:39.862615 815 external_mini_cluster.cc:1428] Setting key c172ae19ae32b528d31a3cf0ea42141f
I20260619 12:00:39.863210 11129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 696e1521635149eabba43ae4f9c7d7be. Latest consensus state: current_term: 1 leader_uuid: "696e1521635149eabba43ae4f9c7d7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696e1521635149eabba43ae4f9c7d7be" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35901 } } }
I20260619 12:00:39.863368 11129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:39.864157 11140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:00:39.863868 11128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "696e1521635149eabba43ae4f9c7d7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696e1521635149eabba43ae4f9c7d7be" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35901 } } }
I20260619 12:00:39.864316 11128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be [sys.catalog]: This master's current role is: LEADER
I20260619 12:00:39.864831 11140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:00:39.866995 11140 catalog_manager.cc:1347] Generated new cluster ID: 6db54d51c15d4373a1c6af1a800b0b32
I20260619 12:00:39.867065 11140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:00:39 dist-test-slave-fbq8 krb5kdc[11043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:39.875183 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:39.863319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.0.203.254:35901)
0619 12:00:39.863748 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:39.863762 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:39.863805 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:00:39.863949 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:39.863957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:39.864036 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:39.864193 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:39.865118 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:39.866566 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:39.867617 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:39.868116 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:39.868244 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:39.871011 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:39.871017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:39.871017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:39.871033 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:39.873520 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:39.873903 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:39.873907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:39.873907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:39.874017 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:39.874235 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:39.874236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:39.874237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:39.874516 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:39.874702 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:39.874797 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 12:00:39.874954 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":105,"threads_started":1}
I20260619 12:00:39.886678 11140 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:00:39.887456 11140 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:00:39.892550 11140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be: Generated new TSK 0
I20260619 12:00:39.892827 11140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:00:39.927343 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35901
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:40.047715 11150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:40.047945 11150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:40.047972 11150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:40.049261 11150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:40.049315 11150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:40.049345 11150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:00:40.050884 11150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:40.051167 11150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:40.051441 11150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:40.058066 11155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:40.058135 11156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:40.057902 11158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:40.058074 11150 server_base.cc:1034] running on GCE node
I20260619 12:00:40.058519 11150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:40.058815 11150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:40.060027 11150 hybrid_clock.cc:648] HybridClock initialized: now 1781870440060011 us; error 50 us; skew 500 ppm
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:40.064185 11150 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:00:40.064821 11150 webserver.cc:460] Webserver started at http://127.0.203.193:43501/ using document root <none> and password file <none>
I20260619 12:00:40.065104 11150 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:40.065176 11150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:40.065335 11150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:40.066457 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "45a5d79e90ed4e3681afb24c3f668b28"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "220a2d292de81e9b9fcfb83762d2e3a0"
server_key_iv: "f07433edecb331692d8d91e1829b722d"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.066823 11150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "45a5d79e90ed4e3681afb24c3f668b28"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "220a2d292de81e9b9fcfb83762d2e3a0"
server_key_iv: "f07433edecb331692d8d91e1829b722d"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.068282 11150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:00:40.069270 11165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.069593 11150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:00:40.069707 11150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "45a5d79e90ed4e3681afb24c3f668b28"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "220a2d292de81e9b9fcfb83762d2e3a0"
server_key_iv: "f07433edecb331692d8d91e1829b722d"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.069795 11150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:40.080117 11150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:40.083189 11150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:40.083398 11150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:40.083668 11150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:40.084013 11150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:40.084070 11150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.084115 11150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:40.084148 11150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.091730 11150 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:40861
I20260619 12:00:40.091784 11278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:40861 every 8 connection(s)
I20260619 12:00:40.092213 11150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:00:40.095090 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 11150
I20260619 12:00:40.095242 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:00:40.095458 815 external_mini_cluster.cc:1428] Setting key 0820070307c234b1b5e5921d48f8c98a
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:40.104482 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.093089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:40547 (local address 127.0.203.254:35901)
0619 12:00:40.093136 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.093138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.093480 (+ 342us) server_negotiation.cc:407] Connection header received
0619 12:00:40.093972 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.093974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.093988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.094054 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:40.095156 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.095850 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.097403 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.097793 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.097871 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.101096 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:40.101099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:40.101099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:40.101108 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:40.102874 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.103398 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.103399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.103400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.103467 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.103739 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.103740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.103740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.103875 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:40.103955 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.104260 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.104381 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 12:00:40.104769 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.093065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35901 (local address 127.0.203.193:40547)
0619 12:00:40.093261 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.093315 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.093831 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.094137 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.094141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.094763 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:40.095029 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.095034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.096023 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.096026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.097285 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.097289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.097901 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.097902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.098088 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.098771 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:40.098785 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:40.100986 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:40.103008 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:40.103021 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:40.103024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:40.103310 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:40.103541 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:40.103542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:40.103542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:40.103669 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:40.103951 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:40.103963 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:40.104122 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.104238 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.104430 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 12:00:40.105319 11279 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35901
I20260619 12:00:40.105422 11279 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:40.105641 11279 heartbeater.cc:511] Master 127.0.203.254:35901 requested a full tablet report, sending...
I20260619 12:00:40.106213 11088 ts_manager.cc:194] Registered new tserver with Master: 45a5d79e90ed4e3681afb24c3f668b28 (127.0.203.193:40861)
I20260619 12:00:40.106878 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:40547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:00:40.132555 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35901
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:40.228156 11284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:40.228442 11284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:40.228482 11284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:40.229859 11284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:40.229941 11284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:40.229993 11284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:00:40.231546 11284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:40.231851 11284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:40.232108 11284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:40.237910 11290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:40.237877 11292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:40.237923 11289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:40.238354 11284 server_base.cc:1034] running on GCE node
I20260619 12:00:40.238624 11284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:40.238916 11284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:40.240101 11284 hybrid_clock.cc:648] HybridClock initialized: now 1781870440240076 us; error 44 us; skew 500 ppm
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:40.243098 11284 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:00:40.243647 11284 webserver.cc:460] Webserver started at http://127.0.203.194:42557/ using document root <none> and password file <none>
I20260619 12:00:40.243927 11284 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:40.244010 11284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:40.244181 11284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:40.245436 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "2bebf5f250aa4eebbbc8acca88117136"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "4bcc5ce56e935177f1ae9c8fa3365838"
server_key_iv: "8b45ef258cd0032c51ad792d761df976"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.245846 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "2bebf5f250aa4eebbbc8acca88117136"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "4bcc5ce56e935177f1ae9c8fa3365838"
server_key_iv: "8b45ef258cd0032c51ad792d761df976"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.247466 11284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 12:00:40.248389 11299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.248791 11284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:40.248905 11284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "2bebf5f250aa4eebbbc8acca88117136"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "4bcc5ce56e935177f1ae9c8fa3365838"
server_key_iv: "8b45ef258cd0032c51ad792d761df976"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.248996 11284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:40.258265 11284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:40.260919 11284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:40.261091 11284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:40.261355 11284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:40.261767 11284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:40.261824 11284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.261873 11284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:40.261895 11284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.269749 11284 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:37899
I20260619 12:00:40.269906 11412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:37899 every 8 connection(s)
I20260619 12:00:40.270195 11284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:40.279112 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 11284
I20260619 12:00:40.279258 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:00:40.279537 815 external_mini_cluster.cc:1428] Setting key 61e676cf44b97b5ddb84b6a5891c7212
I20260619 12:00:40.282142 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.270982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35901 (local address 127.0.203.194:35117)
0619 12:00:40.271182 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.271227 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.271750 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.271993 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.271996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.272610 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:40.272919 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.272924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.273802 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.273804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.274752 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.274756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.275177 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.275178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.275320 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.276188 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:40.276198 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:40.277925 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:40.280083 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:40.280097 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:40.280101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:40.280581 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:40.280911 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:40.280911 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:40.280912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:40.281030 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:40.281531 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:40.281547 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:40.281693 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.281797 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.281957 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
I20260619 12:00:40.282141 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.271038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:35117 (local address 127.0.203.254:35901)
0619 12:00:40.271106 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.271109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.271339 (+ 230us) server_negotiation.cc:407] Connection header received
0619 12:00:40.271844 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.271845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.271856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.271936 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:40.273051 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.273670 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.274869 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.275070 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.275157 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.278035 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:40.278036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:40.278037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:40.278045 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:40.279918 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.280741 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.280743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.280743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.280817 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.281122 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.281123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.281123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.281384 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:40.281480 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.281865 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.281982 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:00:40.282716 11413 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35901
I20260619 12:00:40.282858 11413 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:40.283108 11413 heartbeater.cc:511] Master 127.0.203.254:35901 requested a full tablet report, sending...
I20260619 12:00:40.283618 11088 ts_manager.cc:194] Registered new tserver with Master: 2bebf5f250aa4eebbbc8acca88117136 (127.0.203.194:37899)
I20260619 12:00:40.284176 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:35117
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:00:40.311579 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35901
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:00:40.405339 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:00:40.405587 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:00:40.405615 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:00:40.406739 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:00:40.406783 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:00:40.406811 11418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:00:40.408331 11418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:00:40.408552 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:00:40.408819 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:00:40.415216 11424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:00:40.415238 11423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:40.415692 11418 server_base.cc:1034] running on GCE node
W20260619 12:00:40.415238 11426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:00:40.416059 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:00:40.416361 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:00:40.417537 11418 hybrid_clock.cc:648] HybridClock initialized: now 1781870440417528 us; error 34 us; skew 500 ppm
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:00:40.420535 11418 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:00:40.421165 11418 webserver.cc:460] Webserver started at http://127.0.203.195:37331/ using document root <none> and password file <none>
I20260619 12:00:40.421422 11418 fs_manager.cc:359] Metadata directory not provided
I20260619 12:00:40.421494 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:00:40.421653 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:00:40.422842 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "41fae43c3f484343a6902412d1248b67"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "3b26a2928f36d53bb98a4c1db28f0290"
server_key_iv: "ff3bc2de8ecc21272d85bf8b00f6cadb"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.423265 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "41fae43c3f484343a6902412d1248b67"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "3b26a2928f36d53bb98a4c1db28f0290"
server_key_iv: "ff3bc2de8ecc21272d85bf8b00f6cadb"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.425020 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 12:00:40.425994 11433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.426314 11418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:00:40.426440 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "41fae43c3f484343a6902412d1248b67"
format_stamp: "Formatted at 2026-06-19 12:00:40 on dist-test-slave-fbq8"
server_key: "3b26a2928f36d53bb98a4c1db28f0290"
server_key_iv: "ff3bc2de8ecc21272d85bf8b00f6cadb"
server_key_version: "encryptionkey@0"
I20260619 12:00:40.426523 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:00:40.436370 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:00:40.439067 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:00:40.439215 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:00:40.439450 11418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:00:40.439836 11418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:00:40.439875 11418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.439903 11418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:00:40.439960 11418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:00:40.448407 11418 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:46343
I20260619 12:00:40.448490 11546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:46343 every 8 connection(s)
I20260619 12:00:40.448931 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:00:40.458256 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 11418
I20260619 12:00:40.458389 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:00:40.458617 815 external_mini_cluster.cc:1428] Setting key 110c88b8a51cff1193a0663798a528ba
I20260619 12:00:40.460525 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.449923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:37223 (local address 127.0.203.254:35901)
0619 12:00:40.449979 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.449985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.450219 (+ 234us) server_negotiation.cc:407] Connection header received
0619 12:00:40.450701 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.450702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.450716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.450810 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:40.451934 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.452558 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.453840 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.454060 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.454143 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.456900 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:40.456902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:40.456902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:40.456910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:40.458734 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.459359 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.459360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.459361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.459428 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.459741 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.459744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.459744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.459940 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:40.460028 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.460318 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.460426 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 12:00:40.460747 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.449817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35901 (local address 127.0.203.195:37223)
0619 12:00:40.450002 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.450046 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.450575 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.450946 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.450950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.451570 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:00:40.451786 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.451791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.452745 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.452746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.453738 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.453741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.454189 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.454191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.454329 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.455016 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:00:40.455025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:00:40.456724 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:00:40.458875 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:40.458915 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:40.458919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:40.459265 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:40.459534 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:00:40.459539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:00:40.459539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:00:40.459653 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:00:40.460023 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:00:40.460030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:00:40.460180 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.460294 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.460461 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
I20260619 12:00:40.461280 11547 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35901
I20260619 12:00:40.461412 11547 heartbeater.cc:464] Registering TS with master...
I20260619 12:00:40.461648 11547 heartbeater.cc:511] Master 127.0.203.254:35901 requested a full tablet report, sending...
I20260619 12:00:40.462116 11088 ts_manager.cc:194] Registered new tserver with Master: 41fae43c3f484343a6902412d1248b67 (127.0.203.195:46343)
I20260619 12:00:40.462555 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:37223
I20260619 12:00:40.471128 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:00:40.479981 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.472826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.0.203.254:35901)
0619 12:00:40.472933 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.472935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.473017 (+ 82us) server_negotiation.cc:407] Connection header received
0619 12:00:40.473101 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.473102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.473113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.473157 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:00:40.473886 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.474546 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.475380 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.475578 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.475649 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.477046 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:00:40.477047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:00:40.477048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:00:40.477055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:00:40.478582 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.478944 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.478945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.478945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.479000 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:00:40.479231 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:00:40.479232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:00:40.479232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:00:40.479397 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:00:40.479502 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.479736 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.479826 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:00:40.482321 11088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:00:40.484253 11088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:00:40.491909 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.487566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:46343 (local address 127.0.0.1:44036)
0619 12:00:40.487794 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.487807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.487870 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.488256 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.488260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.488263 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:40.488358 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.488361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.489505 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.489506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.490548 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.490550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.491579 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.491581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.491652 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.491662 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.491749 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.491820 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":158,"threads_started":1}
I20260619 12:00:40.491912 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.487677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.0.203.195:46343)
0619 12:00:40.488048 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.488054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.488065 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:40.488112 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.488114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.488133 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.488251 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:40.488456 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.489383 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.490654 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.491400 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.491477 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.491539 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.491767 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.491812 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":132,"threads_started":1}
I20260619 12:00:40.492684 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.487543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37899 (local address 127.0.0.1:41268)
0619 12:00:40.487729 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.487781 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.487865 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.488701 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.488702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.488706 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:40.488757 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.488759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.490217 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.490218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.491406 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.491410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.492335 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.492337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.492417 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.492424 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.492484 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.492536 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":7,"thread_start_us":111,"threads_started":1}
I20260619 12:00:40.492684 11561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.487855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40861 (local address 127.0.0.1:57530)
0619 12:00:40.488080 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.488090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.488136 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.488742 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.488743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.488747 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:40.488814 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.488816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.490201 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.490202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.491287 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.491290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.492337 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.492338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.492417 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.492424 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.492486 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.492536 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":131,"thread_start_us":85,"threads_started":1}
I20260619 12:00:40.492996 11481 tablet_service.cc:1467] Processing CreateTablet for tablet a9b857dced624ead9d900431007c625e (DEFAULT_TABLE table=test-table [id=9d239919bae142a98eda06d933031ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:40.493379 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b857dced624ead9d900431007c625e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:40.496016 11564 tablet_bootstrap.cc:492] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Bootstrap starting.
I20260619 12:00:40.496188 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.487949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.0.203.194:37899)
0619 12:00:40.488392 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.488396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.488406 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:00:40.488447 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.488450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.488468 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.488552 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:40.488849 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.490074 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.491496 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.492174 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.492237 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.492294 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.495989 (+ 3695us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.496075 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":127,"threads_started":1}
I20260619 12:00:40.496505 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.488080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.0.203.193:40861)
0619 12:00:40.488544 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.488548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.488559 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:00:40.488601 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.488603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.488620 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.488738 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:40.488916 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.490054 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.491384 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.492151 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.492233 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.492293 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.496331 (+ 4038us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.496405 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":111,"threads_started":1}
I20260619 12:00:40.496999 11564 tablet_bootstrap.cc:654] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:40.497067 11347 tablet_service.cc:1467] Processing CreateTablet for tablet a9b857dced624ead9d900431007c625e (DEFAULT_TABLE table=test-table [id=9d239919bae142a98eda06d933031ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:40.497217 11213 tablet_service.cc:1467] Processing CreateTablet for tablet a9b857dced624ead9d900431007c625e (DEFAULT_TABLE table=test-table [id=9d239919bae142a98eda06d933031ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:00:40.497543 11347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b857dced624ead9d900431007c625e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:40.497591 11213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9b857dced624ead9d900431007c625e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:00:40.497725 11564 log.cc:824] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:40.498551 11564 tablet_bootstrap.cc:492] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: No bootstrap required, opened a new log
I20260619 12:00:40.498633 11564 ts_tablet_manager.cc:1397] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:00:40.500572 11566 tablet_bootstrap.cc:492] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28: Bootstrap starting.
I20260619 12:00:40.500967 11564 raft_consensus.cc:348] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.501188 11564 raft_consensus.cc:374] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:40.501243 11564 raft_consensus.cc:729] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fae43c3f484343a6902412d1248b67, State: Initialized, Role: FOLLOWER
I20260619 12:00:40.501436 11566 tablet_bootstrap.cc:654] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:40.501372 11564 consensus_queue.cc:260] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.501675 11567 tablet_bootstrap.cc:492] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Bootstrap starting.
I20260619 12:00:40.501724 11564 ts_tablet_manager.cc:1428] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260619 12:00:40.501919 11547 heartbeater.cc:503] Master 127.0.203.254:35901 was elected leader, sending a full tablet report...
I20260619 12:00:40.502072 11566 log.cc:824] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:40.502377 11567 tablet_bootstrap.cc:654] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Neither blocks nor log segments found. Creating new log.
I20260619 12:00:40.502983 11566 tablet_bootstrap.cc:492] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28: No bootstrap required, opened a new log
I20260619 12:00:40.503046 11567 log.cc:824] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Log is configured to *not* fsync() on all Append() calls
I20260619 12:00:40.503072 11566 ts_tablet_manager.cc:1397] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:00:40.503906 11567 tablet_bootstrap.cc:492] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: No bootstrap required, opened a new log
I20260619 12:00:40.503969 11567 ts_tablet_manager.cc:1397] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:00:40.504798 11566 raft_consensus.cc:348] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.504964 11566 raft_consensus.cc:374] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:40.505115 11566 raft_consensus.cc:729] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a5d79e90ed4e3681afb24c3f668b28, State: Initialized, Role: FOLLOWER
I20260619 12:00:40.505278 11566 consensus_queue.cc:260] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.505618 11566 ts_tablet_manager.cc:1428] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 12:00:40.505685 11279 heartbeater.cc:503] Master 127.0.203.254:35901 was elected leader, sending a full tablet report...
I20260619 12:00:40.505934 11567 raft_consensus.cc:348] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.506233 11567 raft_consensus.cc:374] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:00:40.506274 11567 raft_consensus.cc:729] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bebf5f250aa4eebbbc8acca88117136, State: Initialized, Role: FOLLOWER
I20260619 12:00:40.506381 11567 consensus_queue.cc:260] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.506731 11567 ts_tablet_manager.cc:1428] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:00:40.506834 11413 heartbeater.cc:503] Master 127.0.203.254:35901 was elected leader, sending a full tablet report...
W20260619 12:00:40.520895 11414 tablet.cc:2369] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:00:40.593093 11280 tablet.cc:2369] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:00:40.659706 11571 raft_consensus.cc:482] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:00:40.659904 11571 raft_consensus.cc:504] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.660295 11571 leader_election.cc:288] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41fae43c3f484343a6902412d1248b67 (127.0.203.195:46343), 2bebf5f250aa4eebbbc8acca88117136 (127.0.203.194:37899)
I20260619 12:00:40.664461 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.660836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:34123 (local address 127.0.203.194:37899)
0619 12:00:40.660905 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.660907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.660926 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:00:40.660953 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.660954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.660962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.661051 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:40.661432 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.662198 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.663568 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.664069 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.664162 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.664172 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.664342 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.664381 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:00:40.664523 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.660727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:51265 (local address 127.0.203.195:46343)
0619 12:00:40.660779 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.660781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.661088 (+ 307us) server_negotiation.cc:407] Connection header received
0619 12:00:40.661113 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.661114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.661122 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.661206 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:00:40.661432 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.662190 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.663598 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.664066 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.664183 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.664195 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.664385 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.664433 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:00:40.664535 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.660580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37899 (local address 127.0.203.193:34123)
0619 12:00:40.660780 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.660794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.660868 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.661103 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.661105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.661108 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:40.661250 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.661254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.662340 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.662342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.663479 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.663483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.664162 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.664163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.664261 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.664267 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.664336 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.664390 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":40,"thread_start_us":118,"threads_started":1}
I20260619 12:00:40.664548 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.660578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:46343 (local address 127.0.203.193:51265)
0619 12:00:40.660780 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.660794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:00:40.660871 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:00:40.661234 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:00:40.661235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:00:40.661237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:00:40.661311 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.661312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.662363 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.662364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.663488 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:00:40.663491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.664182 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.664183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:00:40.664293 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.664297 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:00:40.664375 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:00:40.664433 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":28,"thread_start_us":117,"threads_started":1}
I20260619 12:00:40.665004 11367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b857dced624ead9d900431007c625e" candidate_uuid: "45a5d79e90ed4e3681afb24c3f668b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bebf5f250aa4eebbbc8acca88117136" is_pre_election: true
I20260619 12:00:40.665185 11367 raft_consensus.cc:2454] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a5d79e90ed4e3681afb24c3f668b28 in term 0.
I20260619 12:00:40.665416 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b857dced624ead9d900431007c625e" candidate_uuid: "45a5d79e90ed4e3681afb24c3f668b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fae43c3f484343a6902412d1248b67" is_pre_election: true
I20260619 12:00:40.665488 11166 leader_election.cc:302] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bebf5f250aa4eebbbc8acca88117136, 45a5d79e90ed4e3681afb24c3f668b28; no voters:
I20260619 12:00:40.665580 11501 raft_consensus.cc:2454] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45a5d79e90ed4e3681afb24c3f668b28 in term 0.
I20260619 12:00:40.665705 11571 raft_consensus.cc:2781] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:00:40.665795 11571 raft_consensus.cc:482] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:00:40.665833 11571 raft_consensus.cc:3037] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:40.666603 11571 raft_consensus.cc:504] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.666770 11571 leader_election.cc:288] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [CANDIDATE]: Term 1 election: Requested vote from peers 41fae43c3f484343a6902412d1248b67 (127.0.203.195:46343), 2bebf5f250aa4eebbbc8acca88117136 (127.0.203.194:37899)
I20260619 12:00:40.666994 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b857dced624ead9d900431007c625e" candidate_uuid: "45a5d79e90ed4e3681afb24c3f668b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fae43c3f484343a6902412d1248b67"
I20260619 12:00:40.666997 11367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9b857dced624ead9d900431007c625e" candidate_uuid: "45a5d79e90ed4e3681afb24c3f668b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bebf5f250aa4eebbbc8acca88117136"
I20260619 12:00:40.667088 11501 raft_consensus.cc:3037] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:40.667102 11367 raft_consensus.cc:3037] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:00:40.667829 11501 raft_consensus.cc:2454] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a5d79e90ed4e3681afb24c3f668b28 in term 1.
I20260619 12:00:40.668025 11167 leader_election.cc:302] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41fae43c3f484343a6902412d1248b67, 45a5d79e90ed4e3681afb24c3f668b28; no voters:
I20260619 12:00:40.668138 11571 raft_consensus.cc:2781] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:00:40.668213 11367 raft_consensus.cc:2454] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a5d79e90ed4e3681afb24c3f668b28 in term 1.
I20260619 12:00:40.668308 11571 raft_consensus.cc:686] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [term 1 LEADER]: Becoming Leader. State: Replica: 45a5d79e90ed4e3681afb24c3f668b28, State: Running, Role: LEADER
I20260619 12:00:40.668533 11571 consensus_queue.cc:237] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } }
I20260619 12:00:40.669670 11088 catalog_manager.cc:5495] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45a5d79e90ed4e3681afb24c3f668b28 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "45a5d79e90ed4e3681afb24c3f668b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a5d79e90ed4e3681afb24c3f668b28" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40861 } health_report { overall_health: HEALTHY } } }
I20260619 12:00:40.677516 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.674810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.203.193:40861)
0619 12:00:40.674855 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 12:00:40.674857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:00:40.674870 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:00:40.674954 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:00:40.674954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:00:40.674970 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:00:40.675032 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:00:40.675215 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.675714 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.676764 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:00:40.676952 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:00:40.677012 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:00:40.677192 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:00:40.677305 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:00:40.677385 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:00:40.677431 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 12:00:40.677450 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 12:00:40.679479 11501 raft_consensus.cc:1261] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 1 FOLLOWER]: Refusing update from remote peer 45a5d79e90ed4e3681afb24c3f668b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:40.679494 11367 raft_consensus.cc:1261] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 1 FOLLOWER]: Refusing update from remote peer 45a5d79e90ed4e3681afb24c3f668b28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:00:40.679898 11575 consensus_queue.cc:1035] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41fae43c3f484343a6902412d1248b67" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:40.680039 11571 consensus_queue.cc:1035] T a9b857dced624ead9d900431007c625e P 45a5d79e90ed4e3681afb24c3f668b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bebf5f250aa4eebbbc8acca88117136" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:00:40.683441 11580 mvcc.cc:205] Tried to move back new op lower bound from 7298541325020200960 to 7298541324979113984. Current Snapshot: MvccSnapshot[applied={T|T < 7298541325020200960}]
I20260619 12:00:40.684798 11579 mvcc.cc:205] Tried to move back new op lower bound from 7298541325020200960 to 7298541324979113984. Current Snapshot: MvccSnapshot[applied={T|T < 7298541325020200960 or (T in {7298541325020200960})}]
I20260619 12:00:40.685185 11087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:40.685284 11087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:00:40.686157 11087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 696e1521635149eabba43ae4f9c7d7be: Sending DeleteTablet for 3 replicas of tablet a9b857dced624ead9d900431007c625e
I20260619 12:00:40.686534 11481 tablet_service.cc:1514] Processing DeleteTablet for tablet a9b857dced624ead9d900431007c625e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:40 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44036
I20260619 12:00:40.686585 11213 tablet_service.cc:1514] Processing DeleteTablet for tablet a9b857dced624ead9d900431007c625e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:40 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:57530
I20260619 12:00:40.686585 11347 tablet_service.cc:1514] Processing DeleteTablet for tablet a9b857dced624ead9d900431007c625e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:00:40 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:41268
I20260619 12:00:40.686908 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 11150
I20260619 12:00:40.688871 11588 tablet_replica.cc:330] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: stopping tablet replica
I20260619 12:00:40.688972 11588 raft_consensus.cc:2229] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:40.689072 11588 raft_consensus.cc:2258] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:40.689281 11590 tablet_replica.cc:330] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: stopping tablet replica
I20260619 12:00:40.689384 11590 raft_consensus.cc:2229] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:00:40.689492 11590 raft_consensus.cc:2258] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:00:40.689616 11588 ts_tablet_manager.cc:1905] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:40.689936 11590 ts_tablet_manager.cc:1905] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:00:40.691737 11590 ts_tablet_manager.cc:1918] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:40.691824 11588 ts_tablet_manager.cc:1918] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:00:40.691836 11590 log.cc:1196] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/a9b857dced624ead9d900431007c625e
I20260619 12:00:40.691978 11588 log.cc:1196] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/a9b857dced624ead9d900431007c625e
I20260619 12:00:40.692224 11590 ts_tablet_manager.cc:1939] T a9b857dced624ead9d900431007c625e P 2bebf5f250aa4eebbbc8acca88117136: Deleting consensus metadata
I20260619 12:00:40.692376 11588 ts_tablet_manager.cc:1939] T a9b857dced624ead9d900431007c625e P 41fae43c3f484343a6902412d1248b67: Deleting consensus metadata
I20260619 12:00:40.692741 11072 catalog_manager.cc:4841] TS 2bebf5f250aa4eebbbc8acca88117136 (127.0.203.194:37899): tablet a9b857dced624ead9d900431007c625e (table test-table [id=9d239919bae142a98eda06d933031ae7]) successfully deleted
I20260619 12:00:40.692868 11073 catalog_manager.cc:4841] TS 41fae43c3f484343a6902412d1248b67 (127.0.203.195:46343): tablet a9b857dced624ead9d900431007c625e (table test-table [id=9d239919bae142a98eda06d933031ae7]) successfully deleted
W20260619 12:00:40.694008 11073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:40861 (error 108)
I20260619 12:00:40.694572 11561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:00:40.694397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40861 (local address 127.0.0.1:57546)
0619 12:00:40.694484 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 12:00:40.694530 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 12:00:40.694608 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 11284
W20260619 12:00:40.696033 11073 catalog_manager.cc:4584] TS 45a5d79e90ed4e3681afb24c3f668b28 (127.0.203.193:40861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9b857dced624ead9d900431007c625e: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40861: connect: Connection refused (error 111)
I20260619 12:00:40.700088 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 11418
I20260619 12:00:40.705965 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 11056
2026-06-19T12:00:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2182 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11594](info): set up 2 sockets
Jun 19 12:00:40 dist-test-slave-fbq8 krb5kdc[11594](info): commencing operation
krb5kdc: starting...
W20260619 12:00:41.777719 815 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:00:41 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:00:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/rangeradmin_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/rangeradmin_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/rangerlookup_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/rangerlookup_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
I20260619 12:00:41.967509 815 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwGpnVV/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/postgres -l logfile start
2026-06-19 12:00:44.855 UTC [11624] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 12:00:44.855 UTC [11624] LOG: listening on IPv4 address "127.0.203.212", port 42305
2026-06-19 12:00:44.860 UTC [11624] LOG: listening on Unix socket "/tmp/.s.PGSQL.42305"
2026-06-19 12:00:44.897 UTC [11627] LOG: database system was shut down at 2026-06-19 12:00:43 UTC
2026-06-19 12:00:44.901 UTC [11624] LOG: database system is ready to accept connections
I20260619 12:00:45.865526 815 mini_postgres.cc:96] Postgres bound to 42305
2026-06-19 12:00:45.871 UTC [11635] FATAL: database "slave" does not exist
127.0.203.212:42305 - accepting connections
I20260619 12:00:45.872313 815 mini_ranger.cc:161] Starting Ranger...
I20260619 12:00:45.888761 815 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 12:00:46.481025 815 mini_ranger.cc:87] Created ranger Postgres database
I20260619 12:00:46.481127 815 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-admin
2026-06-19 12:00:46,791 [I] DB FLAVOR :POSTGRES
2026-06-19 12:00:46,791 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:00:46,791 [I] Checking connection..
2026-06-19 12:00:46,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 12:00:47,081 [I] Checking connection passed.
2026-06-19 12:00:47,081 [I] --------- Verifying version history table ---------
2026-06-19 12:00:47,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:00:47,339 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:00:47,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:00:47,625 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:00:47,625 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 12:00:47,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 12:00:47.858 UTC [11699] WARNING: there is no transaction in progress
2026-06-19 12:00:47,865 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 12:00:47,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:00:48,142 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 12:00:48,142 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:00:48,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 12:00:48,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 12:00:48,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fbq8.c.gcp-upstream.internal','N') ;"
2026-06-19 12:00:48,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:00:49,212 [I] Table x_portal_user does not exist in database ranger
2026-06-19 12:00:49,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:00:49,478 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 12:00:49,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:00:49,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:00:49,989 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 12:00:49,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 12:00:51.083 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.102 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.127 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.145 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.172 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.196 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.273 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.281 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.302 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.314 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.326 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.342 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.355 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.362 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.372 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:51.382 UTC [11807] WARNING: there is no transaction in progress
2026-06-19 12:00:52,183 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 12:00:52,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 12:00:52,440 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 12:00:52,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:00:52,691 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 12:00:52,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:00:52,992 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 12:00:52,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:00:53,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:00:53,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fbq8.c.gcp-upstream.internal';"
2026-06-19 12:00:53,781 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 12:00:53,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 12:00:54,062 [I] DB_PATCHES have already been applied
I20260619 12:00:54.068804 815 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwGpnVV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 12:00:54.069044 815 mini_ranger.cc:193] Using host: 127.0.203.212
I20260619 12:00:54.073262 815 mini_ranger.cc:227] Ranger admin URL: http://127.0.203.212:50147
Jun 19, 2026 12:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:00:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:00:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:00:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 12:00:55.219429 815 mini_ranger.cc:160] Time spent starting Ranger: real 9.347s user 0.000s sys 0.011s
Jun 19, 2026 12:01:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:01:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 12:01:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:01:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:01:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 12:01:17.505054 815 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/rangerkms_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/rangerkms_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.203.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 12:01:17 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870477, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 12:01:17.593843 815 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 12:01:17.603828 815 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 12:01:18.654623 815 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 12:01:18.654752 815 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms
I20260619 12:01:18.654774 815 mini_ranger_kms.cc:227] Using postgres at 127.0.203.212:42305
2026-06-19 12:01:18,882 [I] DB FLAVOR :POSTGRES
2026-06-19 12:01:18,882 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:01:18,883 [I] Checking connection
2026-06-19 12:01:18,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 12:01:19,190 [I] connection success
2026-06-19 12:01:19,191 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 12:01:19,191 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 12:01:19,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 12:01:19,490 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 12:01:19,491 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:01:19,491 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 12:01:19,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:42305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 12:01:19,811 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 12:01:20.427142 815 mini_ranger_kms.cc:310] Created kms service
I20260619 12:01:20.666286 815 mini_ranger_kms.cc:326] Created kudu user
I20260619 12:01:20.728615 815 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 12:01:20 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870477, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:01:21.231420 815 mini_ranger_kms.cc:384] Added ranger policy
I20260619 12:01:21.231559 815 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 12:01:21.231582 815 mini_ranger_kms.cc:240] Using host: 127.0.203.212
I20260619 12:01:21.235636 815 mini_ranger_kms.cc:276] Ranger KMS PID: 11988
I20260619 12:01:21.235826 815 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.203.212:48215
Jun 19, 2026 12:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:01:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:01:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 12:01:23.673261 815 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.079s user 0.001s sys 0.209s
I20260619 12:01:23.673370 815 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 12:01:23.673449 815 mini_ranger_kms.cc:401] 127.0.203.212:48215/kms/v1/keys
Jun 19 12:01:23 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781870483, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 12:01:24 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870483, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.212@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
Jun 19 12:01:24 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870483, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.212@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
Jun 19 12:01:27 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:01:27.493897 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42873
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:42873
--ranger_config_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:48215/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:01:27.596406 12038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:01:27.596830 12038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:01:27.596863 12038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:01:27.598210 12038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:01:27.598258 12038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:01:27.598274 12038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:01:27.598289 12038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:01:27.598305 12038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:01:27.600030 12038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.203.212:48215/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:42873
--ranger_config_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:01:27.600253 12038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:01:27.600538 12038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:01:27.607028 12038 server_base.cc:1034] running on GCE node
W20260619 12:01:27.606984 12043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:01:27.607038 12044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:01:27.606997 12046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:01:27.607703 12038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:01:27.608103 12038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:01:27.609328 12038 hybrid_clock.cc:648] HybridClock initialized: now 1781870487609283 us; error 65 us; skew 500 ppm
Jun 19 12:01:27 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:01:27.615631 12038 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:01:27.616266 12038 webserver.cc:460] Webserver started at http://127.0.203.254:40919/ using document root <none> and password file <none>
I20260619 12:01:27.616600 12038 fs_manager.cc:359] Metadata directory not provided
I20260619 12:01:27.616743 12038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:01:27.616964 12038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:01:27 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:01:27.670058 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "78bc884e07204563b13635c597d55a01"
format_stamp: "Formatted at 2026-06-19 12:01:27 on dist-test-slave-fbq8"
server_key: "f5ecb849db22fe734ffd3067cc09020f"
server_key_iv: "d5202ff02ceb8ae412acb72fabc1e108"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:27.670643 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "78bc884e07204563b13635c597d55a01"
format_stamp: "Formatted at 2026-06-19 12:01:27 on dist-test-slave-fbq8"
server_key: "f5ecb849db22fe734ffd3067cc09020f"
server_key_iv: "d5202ff02ceb8ae412acb72fabc1e108"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:27.672888 12038 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 12:01:27.688242 12053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:27.688499 12038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:01:27.688616 12038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "78bc884e07204563b13635c597d55a01"
format_stamp: "Formatted at 2026-06-19 12:01:27 on dist-test-slave-fbq8"
server_key: "f5ecb849db22fe734ffd3067cc09020f"
server_key_iv: "d5202ff02ceb8ae412acb72fabc1e108"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:27.688752 12038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:01:27.704952 12038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:01:27.707422 12038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:01:27.707605 12038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:01:27.714587 12038 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:42873
I20260619 12:01:27.714635 12105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:42873 every 8 connection(s)
I20260619 12:01:27.715138 12038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:01:27.716050 12106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:01:27.718731 12106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01: Bootstrap starting.
I20260619 12:01:27.719349 12106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01: Neither blocks nor log segments found. Creating new log.
I20260619 12:01:27.719655 12106 log.cc:824] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01: Log is configured to *not* fsync() on all Append() calls
I20260619 12:01:27.720525 12106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01: No bootstrap required, opened a new log
I20260619 12:01:27.723057 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 12038
I20260619 12:01:27.723177 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:01:27.723383 815 external_mini_cluster.cc:1428] Setting key dfc69263f108d45965d71a4de6232825
I20260619 12:01:27.723429 12106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc884e07204563b13635c597d55a01" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42873 } }
I20260619 12:01:27.723665 12106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:01:27.723737 12106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bc884e07204563b13635c597d55a01, State: Initialized, Role: FOLLOWER
I20260619 12:01:27.723945 12106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc884e07204563b13635c597d55a01" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42873 } }
I20260619 12:01:27.724126 12106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:01:27.724241 12106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:01:27.724355 12106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:01:27.725310 12106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc884e07204563b13635c597d55a01" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42873 } }
I20260619 12:01:27.725464 12106 leader_election.cc:302] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78bc884e07204563b13635c597d55a01; no voters:
I20260619 12:01:27.725780 12106 leader_election.cc:288] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:01:27.725916 12111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:01:27.726168 12106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:01:27.726178 12111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [term 1 LEADER]: Becoming Leader. State: Replica: 78bc884e07204563b13635c597d55a01, State: Running, Role: LEADER
I20260619 12:01:27.726420 12111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc884e07204563b13635c597d55a01" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42873 } }
I20260619 12:01:27.726846 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78bc884e07204563b13635c597d55a01. Latest consensus state: current_term: 1 leader_uuid: "78bc884e07204563b13635c597d55a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc884e07204563b13635c597d55a01" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42873 } } }
I20260619 12:01:27.726922 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [sys.catalog]: This master's current role is: LEADER
I20260619 12:01:27.727259 12113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78bc884e07204563b13635c597d55a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc884e07204563b13635c597d55a01" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42873 } } }
I20260619 12:01:27.727375 12113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01 [sys.catalog]: This master's current role is: LEADER
I20260619 12:01:27.727555 12117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:01:27.728233 12117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:01:27.728578 12106 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:01:27.732358 12117 catalog_manager.cc:1347] Generated new cluster ID: a68b755275fa455da7eaeb92532c62c7
I20260619 12:01:27.732533 12117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:01:27 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:01:27.741472 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:27.724068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.0.203.254:42873)
0619 12:01:27.724346 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:27.724360 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:27.724528 (+ 168us) server_negotiation.cc:407] Connection header received
0619 12:01:27.724770 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:27.724780 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:27.724872 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:27.725070 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:01:27.726173 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:27.727923 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:27.729094 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:27.729661 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:27.729729 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:27.734729 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:01:27.734790 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:01:27.734791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:01:27.734890 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:01:27.738388 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:27.738893 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:27.738896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:27.738897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:27.739020 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:27.739234 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:27.739235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:27.739235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:27.739684 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:01:27.739866 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:27.740580 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 12:01:27.741040 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
I20260619 12:01:27.747465 12117 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:01:27.749325 12117 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:01:27.757033 12117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01: Generated new TSK 0
I20260619 12:01:27.757402 12117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:01:28.615660 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:48215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42873
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:01:28.746429 12151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:01:28.746783 12151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:01:28.746831 12151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:01:28.748879 12151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:01:28.748970 12151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:01:28.749017 12151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:01:28.751605 12151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.203.212:48215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:01:28.751912 12151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:01:28.752292 12151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:01:28.758395 12157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:01:28.758535 12159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:01:28.758395 12156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:01:28.760746 12151 server_base.cc:1034] running on GCE node
I20260619 12:01:28.761072 12151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:01:28.761360 12151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:01:28.768857 12151 hybrid_clock.cc:648] HybridClock initialized: now 1781870488768819 us; error 47 us; skew 500 ppm
Jun 19 12:01:28 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:01:28.772354 12151 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:01:28.773147 12151 webserver.cc:460] Webserver started at http://127.0.203.193:44545/ using document root <none> and password file <none>
I20260619 12:01:28.773500 12151 fs_manager.cc:359] Metadata directory not provided
I20260619 12:01:28.773581 12151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:01:28.773720 12151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:01:28 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:01:28.786971 12151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "606bf224216746908e43842e117eba43"
format_stamp: "Formatted at 2026-06-19 12:01:28 on dist-test-slave-fbq8"
server_key: "44b66559b02295599b2e38a716e29bde"
server_key_iv: "606893e16bdf8d3997fbef1552c163f0"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:28.787636 12151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "606bf224216746908e43842e117eba43"
format_stamp: "Formatted at 2026-06-19 12:01:28 on dist-test-slave-fbq8"
server_key: "44b66559b02295599b2e38a716e29bde"
server_key_iv: "606893e16bdf8d3997fbef1552c163f0"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:28.791239 12151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:01:28.807566 12166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:28.807857 12151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:01:28.807981 12151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "606bf224216746908e43842e117eba43"
format_stamp: "Formatted at 2026-06-19 12:01:28 on dist-test-slave-fbq8"
server_key: "44b66559b02295599b2e38a716e29bde"
server_key_iv: "606893e16bdf8d3997fbef1552c163f0"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:28.808091 12151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:01:28.825393 12151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:01:28.828042 12151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:01:28.828255 12151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:01:28.828548 12151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:01:28.829165 12151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:01:28.829258 12151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:28.829306 12151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:01:28.829325 12151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:28.836791 12151 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:37763
I20260619 12:01:28.837263 12281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:37763 every 8 connection(s)
I20260619 12:01:28.837451 12151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
Jun 19 12:01:28 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:01:28.847008 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 12151
I20260619 12:01:28.847175 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:01:28.847419 815 external_mini_cluster.cc:1428] Setting key 6e9c4f739a08bf73b104128d3cc8b1f4
I20260619 12:01:28.849840 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:28.838352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:42873 (local address 127.0.203.193:34617)
0619 12:01:28.838520 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:28.838563 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:28.839418 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:28.840998 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:28.841003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:28.841221 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:01:28.841419 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:28.841423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:28.842225 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:28.842226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:28.843035 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:28.843038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:28.843446 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:28.843448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:28.843581 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:28.844155 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:01:28.844163 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:01:28.845648 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:01:28.847666 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:01:28.847674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:01:28.847676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:01:28.847941 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:01:28.848710 (+ 769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:01:28.848711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:01:28.848712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:01:28.848821 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:01:28.849110 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:01:28.849117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:01:28.849301 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 12:01:28.849412 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 12:01:28.849594 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260619 12:01:28.850603 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:28.838369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:34617 (local address 127.0.203.254:42873)
0619 12:01:28.840776 (+ 2407us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:28.840778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:28.840787 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:01:28.840811 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:28.840812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:28.840823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:28.840907 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:01:28.841533 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:28.842129 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:28.843144 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:28.843366 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:28.843441 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:28.845756 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:01:28.845757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:01:28.845757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:01:28.845763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:01:28.847595 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:28.848021 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:28.848022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:28.848023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:28.848091 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:28.848899 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:28.848900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:28.848900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:28.849024 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:01:28.849120 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:28.850398 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0619 12:01:28.850490 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2384,"thread_start_us":60,"threads_started":1}
I20260619 12:01:28.851258 12282 heartbeater.cc:347] Connected to a master server at 127.0.203.254:42873
I20260619 12:01:28.851356 12282 heartbeater.cc:464] Registering TS with master...
I20260619 12:01:28.851541 12282 heartbeater.cc:511] Master 127.0.203.254:42873 requested a full tablet report, sending...
I20260619 12:01:28.852103 12070 ts_manager.cc:194] Registered new tserver with Master: 606bf224216746908e43842e117eba43 (127.0.203.193:37763)
I20260619 12:01:28.852952 12070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:34617
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:01:28.880391 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:48215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42873
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:01:29.018342 12288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:01:29.018652 12288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:01:29.018695 12288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:01:29.020728 12288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:01:29.020835 12288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:01:29.020910 12288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:01:29.023815 12288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.203.212:48215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:01:29.024154 12288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:01:29.024472 12288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:01:29.031553 12296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:01:29.031919 12295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:01:29.031948 12298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:01:29.033200 12288 server_base.cc:1034] running on GCE node
I20260619 12:01:29.033448 12288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:01:29.033763 12288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:01:29.034968 12288 hybrid_clock.cc:648] HybridClock initialized: now 1781870489034929 us; error 52 us; skew 500 ppm
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:01:29.038576 12288 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:01:29.039208 12288 webserver.cc:460] Webserver started at http://127.0.203.194:34979/ using document root <none> and password file <none>
I20260619 12:01:29.039480 12288 fs_manager.cc:359] Metadata directory not provided
I20260619 12:01:29.039553 12288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:01:29.039700 12288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:01:29.053769 12288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "68ede89a42624419910d3d5f94e2754c"
format_stamp: "Formatted at 2026-06-19 12:01:29 on dist-test-slave-fbq8"
server_key: "8048a4895111c05c3b22a891a2f7bd3a"
server_key_iv: "53ccca2d3b84ef1bcab78e3701ba0b58"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:29.054284 12288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "68ede89a42624419910d3d5f94e2754c"
format_stamp: "Formatted at 2026-06-19 12:01:29 on dist-test-slave-fbq8"
server_key: "8048a4895111c05c3b22a891a2f7bd3a"
server_key_iv: "53ccca2d3b84ef1bcab78e3701ba0b58"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:29.056490 12288 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:01:29.077778 12305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:29.080195 12288 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260619 12:01:29.080345 12288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "68ede89a42624419910d3d5f94e2754c"
format_stamp: "Formatted at 2026-06-19 12:01:29 on dist-test-slave-fbq8"
server_key: "8048a4895111c05c3b22a891a2f7bd3a"
server_key_iv: "53ccca2d3b84ef1bcab78e3701ba0b58"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:29.080473 12288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:01:29.105743 12288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:01:29.108196 12288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:01:29.108354 12288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:01:29.108628 12288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:01:29.109045 12288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:01:29.109097 12288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:29.109133 12288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:01:29.109221 12288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:29.119560 12288 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:44809
I20260619 12:01:29.120110 12288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:01:29.121670 12418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:44809 every 8 connection(s)
I20260619 12:01:29.131031 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 12288
I20260619 12:01:29.131145 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:01:29.131345 815 external_mini_cluster.cc:1428] Setting key aa628ea37b3bea76110882bb88dd9710
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:01:29.136220 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.121391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:43483 (local address 127.0.203.254:42873)
0619 12:01:29.121443 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.121445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.125007 (+ 3562us) server_negotiation.cc:407] Connection header received
0619 12:01:29.125817 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.125819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.125839 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.125871 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:01:29.126798 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.127607 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.129208 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.129492 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.129542 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.132739 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:01:29.132741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:01:29.132741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:01:29.132749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:01:29.134541 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:29.135068 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:29.135069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:29.135070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:29.135128 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:29.135417 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:29.135418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:29.135419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:29.135558 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:01:29.135609 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.135985 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.136094 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 12:01:29.137172 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.121399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:42873 (local address 127.0.203.194:43483)
0619 12:01:29.124806 (+ 3407us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.124866 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:29.125665 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:29.126083 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:29.126089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:29.126381 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:01:29.126689 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.126695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.127732 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.127733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.129069 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.129072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.129640 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.129641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.130236 (+ 595us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.130899 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:01:29.130910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:01:29.132585 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:01:29.134646 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:01:29.134654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:01:29.134657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:01:29.134979 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:01:29.135207 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:01:29.135208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:01:29.135208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:01:29.135342 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:01:29.135641 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:01:29.135646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:01:29.135824 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 12:01:29.136589 (+ 765us) client_negotiation.cc:240] Negotiation successful
0619 12:01:29.136911 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3341,"thread_start_us":91,"threads_started":1}
I20260619 12:01:29.137924 12419 heartbeater.cc:347] Connected to a master server at 127.0.203.254:42873
I20260619 12:01:29.138041 12419 heartbeater.cc:464] Registering TS with master...
I20260619 12:01:29.138254 12419 heartbeater.cc:511] Master 127.0.203.254:42873 requested a full tablet report, sending...
I20260619 12:01:29.138736 12070 ts_manager.cc:194] Registered new tserver with Master: 68ede89a42624419910d3d5f94e2754c (127.0.203.194:44809)
I20260619 12:01:29.139129 12070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:43483
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:01:29.178813 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:48215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42873
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:01:29.313117 12425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:01:29.313524 12425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:01:29.313633 12425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:01:29.315716 12425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:01:29.315852 12425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:01:29.315953 12425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:01:29.318483 12425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.203.212:48215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:42873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:01:29.319357 12425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:01:29.319790 12425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:01:29.332839 12430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:01:29.333431 12433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:01:29.333817 12425 server_base.cc:1034] running on GCE node
W20260619 12:01:29.334081 12431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:01:29.334457 12425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:01:29.334787 12425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:01:29.342149 12425 hybrid_clock.cc:648] HybridClock initialized: now 1781870489342120 us; error 48 us; skew 500 ppm
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:01:29.345723 12425 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:01:29.346400 12425 webserver.cc:460] Webserver started at http://127.0.203.195:39797/ using document root <none> and password file <none>
I20260619 12:01:29.346640 12425 fs_manager.cc:359] Metadata directory not provided
I20260619 12:01:29.346686 12425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:01:29.346809 12425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:01:29.360399 12425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "19072270a5c04deaa8a4ec09d688d190"
format_stamp: "Formatted at 2026-06-19 12:01:29 on dist-test-slave-fbq8"
server_key: "e919d4e3406cd4c1d147293a14dee575"
server_key_iv: "b8681732ea9243307c1d23c20db3fa97"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:29.360863 12425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "19072270a5c04deaa8a4ec09d688d190"
format_stamp: "Formatted at 2026-06-19 12:01:29 on dist-test-slave-fbq8"
server_key: "e919d4e3406cd4c1d147293a14dee575"
server_key_iv: "b8681732ea9243307c1d23c20db3fa97"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:29.362355 12425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:01:29.376328 12440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:29.377064 12425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:01:29.377189 12425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "19072270a5c04deaa8a4ec09d688d190"
format_stamp: "Formatted at 2026-06-19 12:01:29 on dist-test-slave-fbq8"
server_key: "e919d4e3406cd4c1d147293a14dee575"
server_key_iv: "b8681732ea9243307c1d23c20db3fa97"
server_key_version: "kuduclusterkey@0"
I20260619 12:01:29.377312 12425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:01:29.392763 12425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:01:29.395454 12425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:01:29.395618 12425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:01:29.395901 12425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:01:29.396276 12425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:01:29.396327 12425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:29.396364 12425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:01:29.396386 12425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:01:29.403335 12425 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:41059
I20260619 12:01:29.403776 12553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:41059 every 8 connection(s)
I20260619 12:01:29.403822 12425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:01:29.406280 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 12425
I20260619 12:01:29.406394 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:01:29.406602 815 external_mini_cluster.cc:1428] Setting key c333fec96a46feebfb6d03103ef4cf5f
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:01:29.414918 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.405190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:52849 (local address 127.0.203.254:42873)
0619 12:01:29.405242 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.405244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.405490 (+ 246us) server_negotiation.cc:407] Connection header received
0619 12:01:29.406390 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.406392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.406406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.406440 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:01:29.407319 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.407955 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.409038 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.409249 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.409342 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.411535 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:01:29.411536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:01:29.411537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:01:29.411551 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:01:29.413342 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:29.413832 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:29.413834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:29.413834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:29.413913 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:29.414229 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:29.414230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:29.414231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:29.414361 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:01:29.414435 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.414732 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.414822 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:01:29.415141 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.405196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:42873 (local address 127.0.203.195:52849)
0619 12:01:29.405355 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.405398 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:29.406224 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:29.406648 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:29.406653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:29.406920 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:01:29.407219 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.407225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.408054 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.408054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.408929 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.408931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.409351 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.409352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.409483 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.410069 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:01:29.410077 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:01:29.411401 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:01:29.413452 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:01:29.413460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:01:29.413463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:01:29.413732 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:01:29.413988 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:01:29.413989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:01:29.413989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:01:29.414145 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:01:29.414435 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:01:29.414442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:01:29.414597 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 12:01:29.414713 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 12:01:29.414880 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260619 12:01:29.415671 12554 heartbeater.cc:347] Connected to a master server at 127.0.203.254:42873
I20260619 12:01:29.415798 12554 heartbeater.cc:464] Registering TS with master...
I20260619 12:01:29.415992 12554 heartbeater.cc:511] Master 127.0.203.254:42873 requested a full tablet report, sending...
I20260619 12:01:29.416405 12070 ts_manager.cc:194] Registered new tserver with Master: 19072270a5c04deaa8a4ec09d688d190 (127.0.203.195:41059)
I20260619 12:01:29.416956 12070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:52849
I20260619 12:01:29.418373 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:01:29.427568 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.419870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.0.203.254:42873)
0619 12:01:29.419919 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.419921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.420059 (+ 138us) server_negotiation.cc:407] Connection header received
0619 12:01:29.420134 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.420135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.420146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.420201 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:01:29.420951 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.421451 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.422329 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.422573 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.422675 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.424345 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:01:29.424347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:01:29.424347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:01:29.424355 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:01:29.426237 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:29.426666 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:29.426667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:29.426667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:29.426725 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:01:29.426935 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:01:29.426935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:01:29.426936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:01:29.427085 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:01:29.427160 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.427379 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.427465 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:01:29.429955 12070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:01:29.432364 12070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:01:29.465654 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.455754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:41059 (local address 127.0.0.1:36942)
0619 12:01:29.457432 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.457475 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:29.457548 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:29.458048 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:29.458051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:29.458053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:01:29.458126 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.458128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.462064 (+ 3936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.462065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.463289 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.463292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.465328 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.465329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.465413 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.465426 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 12:01:29.465495 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:01:29.465561 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1644,"thread_start_us":75,"threads_started":1}
I20260619 12:01:29.467926 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.456601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.0.203.195:41059)
0619 12:01:29.457683 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.457689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.457698 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:01:29.457736 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.457738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.457754 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.457818 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:01:29.458211 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.461952 (+ 3741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.464379 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.465162 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.465230 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.465293 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.467784 (+ 2491us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.467836 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":122,"threads_started":1}
I20260619 12:01:29.468446 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.456815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.0.203.193:37763)
0619 12:01:29.457931 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.457935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.459397 (+ 1462us) server_negotiation.cc:407] Connection header received
0619 12:01:29.459433 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.459436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.459452 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.459529 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:01:29.459704 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.460764 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.466319 (+ 5555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.467054 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.467126 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.467176 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.468328 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.468377 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":51,"threads_started":1}
I20260619 12:01:29.468981 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.456181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:44809 (local address 127.0.0.1:44058)
0619 12:01:29.460712 (+ 4531us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.460723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:29.460789 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:29.461088 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:29.461088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:29.461091 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:01:29.461154 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.461155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.462401 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.462402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.463503 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.463506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.468699 (+ 5193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.468700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.468784 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.468789 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:01:29.468849 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 12:01:29.468893 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4512,"thread_start_us":61,"threads_started":1}
I20260619 12:01:29.469288 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.456945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.0.203.194:44809)
0619 12:01:29.458007 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.458011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.460855 (+ 2844us) server_negotiation.cc:407] Connection header received
0619 12:01:29.460903 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.460905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.460920 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.461002 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:01:29.461236 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.462303 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.463628 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.464403 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.464477 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.464554 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.469152 (+ 4598us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.469206 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":76,"threads_started":1}
I20260619 12:01:29.469378 12488 tablet_service.cc:1467] Processing CreateTablet for tablet a189732ac07845a4be8e3e1757a2bbde (DEFAULT_TABLE table=test-table [id=4cd6c5b0b5c344599570ff0db83f7c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:01:29.469758 12488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a189732ac07845a4be8e3e1757a2bbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:01:29.469874 12353 tablet_service.cc:1467] Processing CreateTablet for tablet a189732ac07845a4be8e3e1757a2bbde (DEFAULT_TABLE table=test-table [id=4cd6c5b0b5c344599570ff0db83f7c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:01:29.470216 12353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a189732ac07845a4be8e3e1757a2bbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:01:29.474581 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.455973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:37763 (local address 127.0.0.1:53348)
0619 12:01:29.459284 (+ 3311us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.459294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:29.459338 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:29.459593 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:29.459594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:29.459596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:01:29.459638 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.459639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.460857 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.460858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.466218 (+ 5360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.466221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.467342 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.467343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.467407 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.467411 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:01:29.468191 (+ 780us) client_negotiation.cc:240] Negotiation successful
0619 12:01:29.468240 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3294,"thread_start_us":72,"threads_started":1}
I20260619 12:01:29.474606 12573 tablet_bootstrap.cc:492] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Bootstrap starting.
I20260619 12:01:29.475654 12573 tablet_bootstrap.cc:654] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Neither blocks nor log segments found. Creating new log.
I20260619 12:01:29.475598 12216 tablet_service.cc:1467] Processing CreateTablet for tablet a189732ac07845a4be8e3e1757a2bbde (DEFAULT_TABLE table=test-table [id=4cd6c5b0b5c344599570ff0db83f7c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:01:29.475976 12216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a189732ac07845a4be8e3e1757a2bbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:01:29.476006 12573 log.cc:824] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Log is configured to *not* fsync() on all Append() calls
I20260619 12:01:29.481086 12574 tablet_bootstrap.cc:492] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Bootstrap starting.
I20260619 12:01:29.481338 12576 tablet_bootstrap.cc:492] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43: Bootstrap starting.
I20260619 12:01:29.482172 12574 tablet_bootstrap.cc:654] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Neither blocks nor log segments found. Creating new log.
I20260619 12:01:29.482888 12576 tablet_bootstrap.cc:654] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43: Neither blocks nor log segments found. Creating new log.
I20260619 12:01:29.483610 12574 log.cc:824] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Log is configured to *not* fsync() on all Append() calls
I20260619 12:01:29.484000 12576 log.cc:824] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43: Log is configured to *not* fsync() on all Append() calls
I20260619 12:01:29.485056 12573 tablet_bootstrap.cc:492] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: No bootstrap required, opened a new log
I20260619 12:01:29.485148 12573 ts_tablet_manager.cc:1397] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 12:01:29.485682 12576 tablet_bootstrap.cc:492] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43: No bootstrap required, opened a new log
I20260619 12:01:29.485827 12576 ts_tablet_manager.cc:1397] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:01:29.487816 12573 raft_consensus.cc:348] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.488049 12573 raft_consensus.cc:374] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:01:29.488099 12573 raft_consensus.cc:729] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ede89a42624419910d3d5f94e2754c, State: Initialized, Role: FOLLOWER
I20260619 12:01:29.488248 12573 consensus_queue.cc:260] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.488565 12573 ts_tablet_manager.cc:1428] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 12:01:29.488726 12419 heartbeater.cc:503] Master 127.0.203.254:42873 was elected leader, sending a full tablet report...
I20260619 12:01:29.489719 12576 raft_consensus.cc:348] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.489956 12576 raft_consensus.cc:374] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:01:29.490137 12576 raft_consensus.cc:729] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606bf224216746908e43842e117eba43, State: Initialized, Role: FOLLOWER
I20260619 12:01:29.490340 12576 consensus_queue.cc:260] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.490684 12576 ts_tablet_manager.cc:1428] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260619 12:01:29.490844 12282 heartbeater.cc:503] Master 127.0.203.254:42873 was elected leader, sending a full tablet report...
I20260619 12:01:29.492288 12574 tablet_bootstrap.cc:492] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: No bootstrap required, opened a new log
I20260619 12:01:29.492429 12574 ts_tablet_manager.cc:1397] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 12:01:29.495318 12574 raft_consensus.cc:348] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.495594 12574 raft_consensus.cc:374] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:01:29.495684 12574 raft_consensus.cc:729] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19072270a5c04deaa8a4ec09d688d190, State: Initialized, Role: FOLLOWER
I20260619 12:01:29.495891 12574 consensus_queue.cc:260] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.496284 12574 ts_tablet_manager.cc:1428] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 12:01:29.496452 12554 heartbeater.cc:503] Master 127.0.203.254:42873 was elected leader, sending a full tablet report...
I20260619 12:01:29.543015 12580 raft_consensus.cc:482] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:01:29.543190 12580 raft_consensus.cc:504] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.543572 12580 leader_election.cc:288] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19072270a5c04deaa8a4ec09d688d190 (127.0.203.195:41059), 68ede89a42624419910d3d5f94e2754c (127.0.203.194:44809)
I20260619 12:01:29.552114 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.544213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:53383 (local address 127.0.203.194:44809)
0619 12:01:29.544270 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.544272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.544933 (+ 661us) server_negotiation.cc:407] Connection header received
0619 12:01:29.545036 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.545037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.545050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.545083 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:01:29.545306 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.546022 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.551098 (+ 5076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.551687 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.551746 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.551756 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.551982 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.552026 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:01:29.552670 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.543867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:44809 (local address 127.0.203.193:53383)
0619 12:01:29.544867 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.544880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:29.544962 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:29.545150 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:29.545150 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:29.545152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:01:29.545241 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.545243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.546154 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.546155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.547466 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.547470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.551824 (+ 4354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.551825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.551902 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.551907 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:01:29.552492 (+ 585us) client_negotiation.cc:240] Negotiation successful
0619 12:01:29.552546 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":108,"threads_started":1}
I20260619 12:01:29.553196 12357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a189732ac07845a4be8e3e1757a2bbde" candidate_uuid: "606bf224216746908e43842e117eba43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ede89a42624419910d3d5f94e2754c" is_pre_election: true
I20260619 12:01:29.553378 12357 raft_consensus.cc:2454] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606bf224216746908e43842e117eba43 in term 0.
I20260619 12:01:29.553736 12167 leader_election.cc:302] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606bf224216746908e43842e117eba43, 68ede89a42624419910d3d5f94e2754c; no voters:
I20260619 12:01:29.553884 12580 raft_consensus.cc:2781] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:01:29.553922 12580 raft_consensus.cc:482] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:01:29.553943 12580 raft_consensus.cc:3037] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:01:29.554895 12580 raft_consensus.cc:504] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.555044 12580 leader_election.cc:288] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [CANDIDATE]: Term 1 election: Requested vote from peers 19072270a5c04deaa8a4ec09d688d190 (127.0.203.195:41059), 68ede89a42624419910d3d5f94e2754c (127.0.203.194:44809)
I20260619 12:01:29.555423 12357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a189732ac07845a4be8e3e1757a2bbde" candidate_uuid: "606bf224216746908e43842e117eba43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ede89a42624419910d3d5f94e2754c"
I20260619 12:01:29.555508 12357 raft_consensus.cc:3037] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:01:29.556407 12357 raft_consensus.cc:2454] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606bf224216746908e43842e117eba43 in term 1.
I20260619 12:01:29.556723 12167 leader_election.cc:302] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606bf224216746908e43842e117eba43, 68ede89a42624419910d3d5f94e2754c; no voters:
I20260619 12:01:29.556864 12580 raft_consensus.cc:2781] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:01:29.557032 12580 raft_consensus.cc:686] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [term 1 LEADER]: Becoming Leader. State: Replica: 606bf224216746908e43842e117eba43, State: Running, Role: LEADER
I20260619 12:01:29.557137 12580 consensus_queue.cc:237] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } }
I20260619 12:01:29.558020 12070 catalog_manager.cc:5495] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606bf224216746908e43842e117eba43 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "606bf224216746908e43842e117eba43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606bf224216746908e43842e117eba43" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 } health_report { overall_health: UNKNOWN } } }
I20260619 12:01:29.561646 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.544536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:45843 (local address 127.0.203.195:41059)
0619 12:01:29.544588 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.544590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.547681 (+ 3091us) server_negotiation.cc:407] Connection header received
0619 12:01:29.547787 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.547788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.547797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.547826 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:01:29.548024 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.548824 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.560541 (+ 11717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.561207 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.561266 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.561276 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.561520 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.561567 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 12:01:29.561956 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.544357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:41059 (local address 127.0.203.193:45843)
0619 12:01:29.547620 (+ 3263us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.547633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:01:29.547713 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:01:29.547889 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:01:29.547889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:01:29.547891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:01:29.547959 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.547960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.548958 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.548959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.550961 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:01:29.550965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.561348 (+ 10383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.561349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:01:29.561437 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.561441 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:01:29.561806 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 12:01:29.561858 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3241,"thread_start_us":86,"threads_started":1}
I20260619 12:01:29.562542 12508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a189732ac07845a4be8e3e1757a2bbde" candidate_uuid: "606bf224216746908e43842e117eba43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19072270a5c04deaa8a4ec09d688d190" is_pre_election: true
I20260619 12:01:29.562711 12508 raft_consensus.cc:2454] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606bf224216746908e43842e117eba43 in term 0.
I20260619 12:01:29.562813 12507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a189732ac07845a4be8e3e1757a2bbde" candidate_uuid: "606bf224216746908e43842e117eba43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19072270a5c04deaa8a4ec09d688d190"
I20260619 12:01:29.562857 12507 raft_consensus.cc:3037] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:01:29.563804 12507 raft_consensus.cc:2454] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606bf224216746908e43842e117eba43 in term 1.
I20260619 12:01:29.588699 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.582884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.0.203.193:37763)
0619 12:01:29.584180 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0619 12:01:29.584183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:01:29.584198 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:01:29.584254 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:01:29.584255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:01:29.584287 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:01:29.584393 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:01:29.584629 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.585467 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.586686 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:01:29.587006 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:01:29.587073 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:01:29.587292 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:01:29.587442 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:01:29.587541 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:01:29.587821 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 12:01:29.587848 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262}
W20260619 12:01:29.592171 12283 tablet.cc:2369] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:01:29.598163 12357 raft_consensus.cc:1261] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 1 FOLLOWER]: Refusing update from remote peer 606bf224216746908e43842e117eba43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:01:29.598572 12584 consensus_queue.cc:1035] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ede89a42624419910d3d5f94e2754c" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:01:29.600680 12507 raft_consensus.cc:1261] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 1 FOLLOWER]: Refusing update from remote peer 606bf224216746908e43842e117eba43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:01:29.600973 12584 consensus_queue.cc:1035] T a189732ac07845a4be8e3e1757a2bbde P 606bf224216746908e43842e117eba43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19072270a5c04deaa8a4ec09d688d190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:01:29.605017 12587 mvcc.cc:205] Tried to move back new op lower bound from 7298541525389062144 to 7298541525226676224. Current Snapshot: MvccSnapshot[applied={T|T < 7298541525389062144}]
I20260619 12:01:29.605624 12590 mvcc.cc:205] Tried to move back new op lower bound from 7298541525389062144 to 7298541525226676224. Current Snapshot: MvccSnapshot[applied={T|T < 7298541525389062144}]
Jun 19 12:01:29 dist-test-slave-fbq8 krb5kdc[11594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:01:29.608064 12070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:01:29.608179 12070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:01:29.608903 12070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78bc884e07204563b13635c597d55a01: Sending DeleteTablet for 3 replicas of tablet a189732ac07845a4be8e3e1757a2bbde
I20260619 12:01:29.609441 12488 tablet_service.cc:1514] Processing DeleteTablet for tablet a189732ac07845a4be8e3e1757a2bbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:01:29 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:36942
I20260619 12:01:29.609665 12216 tablet_service.cc:1514] Processing DeleteTablet for tablet a189732ac07845a4be8e3e1757a2bbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:01:29 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:53348
I20260619 12:01:29.609925 12353 tablet_service.cc:1514] Processing DeleteTablet for tablet a189732ac07845a4be8e3e1757a2bbde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:01:29 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44058
I20260619 12:01:29.611857 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 12151
I20260619 12:01:29.611835 12588 mvcc.cc:205] Tried to move back new op lower bound from 7298541525389062144 to 7298541525226676224. Current Snapshot: MvccSnapshot[applied={T|T < 7298541525389062144}]
I20260619 12:01:29.613296 12595 tablet_replica.cc:330] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: stopping tablet replica
I20260619 12:01:29.613437 12595 raft_consensus.cc:2229] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:01:29.613507 12595 raft_consensus.cc:2258] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:01:29.613678 12597 tablet_replica.cc:330] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: stopping tablet replica
I20260619 12:01:29.613756 12597 raft_consensus.cc:2229] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:01:29.613863 12597 raft_consensus.cc:2258] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:01:29.614364 12597 ts_tablet_manager.cc:1905] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:01:29.617043 12595 ts_tablet_manager.cc:1905] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:01:29.618873 12597 ts_tablet_manager.cc:1918] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:01:29.618996 12597 log.cc:1196] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/a189732ac07845a4be8e3e1757a2bbde
I20260619 12:01:29.619336 12597 ts_tablet_manager.cc:1939] T a189732ac07845a4be8e3e1757a2bbde P 68ede89a42624419910d3d5f94e2754c: Deleting consensus metadata
I20260619 12:01:29.619915 12054 catalog_manager.cc:4841] TS 68ede89a42624419910d3d5f94e2754c (127.0.203.194:44809): tablet a189732ac07845a4be8e3e1757a2bbde (table test-table [id=4cd6c5b0b5c344599570ff0db83f7c5b]) successfully deleted
I20260619 12:01:29.622567 12595 ts_tablet_manager.cc:1918] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:01:29.622666 12595 log.cc:1196] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/a189732ac07845a4be8e3e1757a2bbde
I20260619 12:01:29.622931 12595 ts_tablet_manager.cc:1939] T a189732ac07845a4be8e3e1757a2bbde P 19072270a5c04deaa8a4ec09d688d190: Deleting consensus metadata
I20260619 12:01:29.623409 12057 catalog_manager.cc:4841] TS 19072270a5c04deaa8a4ec09d688d190 (127.0.203.195:41059): tablet a189732ac07845a4be8e3e1757a2bbde (table test-table [id=4cd6c5b0b5c344599570ff0db83f7c5b]) successfully deleted
W20260619 12:01:29.623840 12057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:37763 (error 108)
I20260619 12:01:29.624760 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:01:29.624610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:37763 (local address 127.0.0.1:53372)
0619 12:01:29.624692 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 12:01:29.624725 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:37763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 12:01:29.624869 12057 catalog_manager.cc:4584] TS 606bf224216746908e43842e117eba43 (127.0.203.193:37763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a189732ac07845a4be8e3e1757a2bbde: Network error: Client connection negotiation failed: client connection to 127.0.203.193:37763: connect: Connection refused (error 111)
I20260619 12:01:29.625074 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 12288
I20260619 12:01:29.639194 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 12425
I20260619 12:01:29.645828 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 12038
I20260619 12:01:29.652940 815 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 12:01:29.998346 815 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 12:01:29.998485 815 mini_ranger.cc:66] Stopping Ranger...
I20260619 12:01:30.343591 815 mini_ranger.cc:68] Stopped Ranger
2026-06-19 12:01:30.343 UTC [11624] LOG: received smart shutdown request
2026-06-19 12:01:30.347 UTC [11624] LOG: background worker "logical replication launcher" (PID 11633) exited with exit code 1
2026-06-19 12:01:30.347 UTC [11628] LOG: shutting down
2026-06-19 12:01:30.563 UTC [11624] LOG: database system is shut down
2026-06-19T12:01:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49970 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:01:30 dist-test-slave-fbq8 krb5kdc[12608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:01:30 dist-test-slave-fbq8 krb5kdc[12608](info): set up 2 sockets
Jun 19 12:01:30 dist-test-slave-fbq8 krb5kdc[12608](info): commencing operation
krb5kdc: starting...
W20260619 12:01:31.740852 815 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:01:31 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:01:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/rangeradmin_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/rangeradmin_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/rangerlookup_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/rangerlookup_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
I20260619 12:01:31.916843 815 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwGpnVV/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/postgres -l logfile start
2026-06-19 12:01:34.851 UTC [12638] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 12:01:34.851 UTC [12638] LOG: listening on IPv4 address "127.0.203.212", port 45723
2026-06-19 12:01:34.856 UTC [12638] LOG: listening on Unix socket "/tmp/.s.PGSQL.45723"
2026-06-19 12:01:34.879 UTC [12641] LOG: database system was shut down at 2026-06-19 12:01:33 UTC
2026-06-19 12:01:34.884 UTC [12638] LOG: database system is ready to accept connections
I20260619 12:01:35.860781 815 mini_postgres.cc:96] Postgres bound to 45723
2026-06-19 12:01:35.867 UTC [12649] FATAL: database "slave" does not exist
127.0.203.212:45723 - accepting connections
I20260619 12:01:35.867879 815 mini_ranger.cc:161] Starting Ranger...
I20260619 12:01:35.884114 815 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 12:01:36.524513 815 mini_ranger.cc:87] Created ranger Postgres database
I20260619 12:01:36.524633 815 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-admin
2026-06-19 12:01:36,825 [I] DB FLAVOR :POSTGRES
2026-06-19 12:01:36,825 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:01:36,825 [I] Checking connection..
2026-06-19 12:01:36,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 12:01:37,074 [I] Checking connection passed.
2026-06-19 12:01:37,074 [I] --------- Verifying version history table ---------
2026-06-19 12:01:37,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:01:37,314 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:01:37,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:01:37,564 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:01:37,564 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 12:01:37,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 12:01:37.814 UTC [12713] WARNING: there is no transaction in progress
2026-06-19 12:01:37,821 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 12:01:37,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:01:38,101 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 12:01:38,101 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:01:38,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 12:01:38,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 12:01:38,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fbq8.c.gcp-upstream.internal','N') ;"
2026-06-19 12:01:38,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:01:39,190 [I] Table x_portal_user does not exist in database ranger
2026-06-19 12:01:39,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:01:39,448 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 12:01:39,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:01:39,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:01:39,978 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 12:01:39,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 12:01:41.081 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.100 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.118 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.137 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.157 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.180 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.255 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.264 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.283 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.297 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.310 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.327 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.339 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.346 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.354 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:41.361 UTC [12821] WARNING: there is no transaction in progress
2026-06-19 12:01:42,122 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 12:01:42,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 12:01:42,404 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 12:01:42,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:01:42,673 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 12:01:42,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:01:42,920 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 12:01:42,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:01:43,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:01:43,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fbq8.c.gcp-upstream.internal';"
2026-06-19 12:01:43,714 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 12:01:43,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 12:01:43,995 [I] DB_PATCHES have already been applied
I20260619 12:01:44.003851 815 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwGpnVV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 12:01:44.003947 815 mini_ranger.cc:193] Using host: 127.0.203.212
I20260619 12:01:44.007902 815 mini_ranger.cc:227] Ranger admin URL: http://127.0.203.212:42519
Jun 19, 2026 12:01:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:01:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:01:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:01:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:01:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:01:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:01:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 12:01:45.141940 815 mini_ranger.cc:160] Time spent starting Ranger: real 9.274s user 0.000s sys 0.011s
Jun 19, 2026 12:02:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:02:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 12:02:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:02:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:02:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 12:02:08.800238 815 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.203.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.203.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/rangerkms_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.203.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/rangerkms_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.203.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.203.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.203.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/HTTP_127.0.203.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 12:02:08 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870528, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 12:02:08.899161 815 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 12:02:08.909931 815 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 12:02:09.944303 815 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 12:02:09.944410 815 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms
I20260619 12:02:09.944434 815 mini_ranger_kms.cc:227] Using postgres at 127.0.203.212:45723
2026-06-19 12:02:10,145 [I] DB FLAVOR :POSTGRES
2026-06-19 12:02:10,145 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:02:10,145 [I] Checking connection
2026-06-19 12:02:10,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 12:02:10,408 [I] connection success
2026-06-19 12:02:10,409 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 12:02:10,409 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 12:02:10,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 12:02:10,672 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 12:02:10,672 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:02:10,673 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 12:02:10,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.203.212:45723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 12:02:10,984 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 12:02:11.496176 815 mini_ranger_kms.cc:310] Created kms service
I20260619 12:02:11.713218 815 mini_ranger_kms.cc:326] Created kudu user
I20260619 12:02:11.765506 815 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 12:02:11 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870528, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:12.211633 815 mini_ranger_kms.cc:384] Added ranger policy
I20260619 12:02:12.211867 815 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwGpnVV/build/release/bin/postgresql.jar:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwGpnVV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 12:02:12.211887 815 mini_ranger_kms.cc:240] Using host: 127.0.203.212
I20260619 12:02:12.215759 815 mini_ranger_kms.cc:276] Ranger KMS PID: 13002
I20260619 12:02:12.215859 815 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.203.212:35441
Jun 19, 2026 12:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:02:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 12:02:14.483462 815 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.584s user 0.000s sys 0.182s
I20260619 12:02:14.483573 815 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 12:02:14.483630 815 mini_ranger_kms.cc:401] 127.0.203.212:35441/kms/v1/keys
Jun 19 12:02:14 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781870534, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 12:02:14 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870534, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.212@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
Jun 19 12:02:15 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870534, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.203.212@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
Jun 19 12:02:17 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:02:17.741806 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44931
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:44931
--ranger_config_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:17.843061 13052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:17.843295 13052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:17.843318 13052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:17.844544 13052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:17.844606 13052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:17.844620 13052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:17.844633 13052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:17.844703 13052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:17.846359 13052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.203.212:35441/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:44931
--ranger_config_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:17.846654 13052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:17.846902 13052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:02:17.853165 13052 server_base.cc:1034] running on GCE node
W20260619 12:02:17.853165 13058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:17.853281 13057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:17.853255 13060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:17.853837 13052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:17.854262 13052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:17.855463 13052 hybrid_clock.cc:648] HybridClock initialized: now 1781870537855439 us; error 50 us; skew 500 ppm
Jun 19 12:02:17 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:17.861760 13052 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:17.862368 13052 webserver.cc:460] Webserver started at http://127.0.203.254:40863/ using document root <none> and password file <none>
I20260619 12:02:17.862625 13052 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:17.862686 13052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:17.862852 13052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:02:17 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:17.910821 13052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "7266775084784ff7809e5ac8a9ee8e1a"
format_stamp: "Formatted at 2026-06-19 12:02:17 on dist-test-slave-fbq8"
server_key: "d971e9afd36138385a913b89c916dcf4"
server_key_iv: "b71ebff5536d295e98aa52b1b87b2a04"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:17.911266 13052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "7266775084784ff7809e5ac8a9ee8e1a"
format_stamp: "Formatted at 2026-06-19 12:02:17 on dist-test-slave-fbq8"
server_key: "d971e9afd36138385a913b89c916dcf4"
server_key_iv: "b71ebff5536d295e98aa52b1b87b2a04"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:17.912868 13052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:02:17.924856 13067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:17.925091 13052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:17.925205 13052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "7266775084784ff7809e5ac8a9ee8e1a"
format_stamp: "Formatted at 2026-06-19 12:02:17 on dist-test-slave-fbq8"
server_key: "d971e9afd36138385a913b89c916dcf4"
server_key_iv: "b71ebff5536d295e98aa52b1b87b2a04"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:17.925323 13052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:17.935659 13052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:17.937845 13052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:17.938022 13052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:17.943018 13052 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:44931
I20260619 12:02:17.943151 13119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:44931 every 8 connection(s)
I20260619 12:02:17.943490 13052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:17.944212 13120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:17.946523 13120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Bootstrap starting.
I20260619 12:02:17.947108 13120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:17.947412 13120 log.cc:824] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:17.948108 13120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: No bootstrap required, opened a new log
I20260619 12:02:17.948966 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13052
I20260619 12:02:17.949079 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:02:17.949268 815 external_mini_cluster.cc:1428] Setting key f35bc385f94b121270bb11a3e33cf6de
I20260619 12:02:17.950768 13120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:17.950932 13120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:17.950994 13120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266775084784ff7809e5ac8a9ee8e1a, State: Initialized, Role: FOLLOWER
I20260619 12:02:17.951167 13120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:17.951269 13120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:02:17.951304 13120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:02:17.951383 13120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:17.952147 13120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:17.952273 13120 leader_election.cc:302] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7266775084784ff7809e5ac8a9ee8e1a; no voters:
I20260619 12:02:17.952483 13120 leader_election.cc:288] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:02:17.952517 13125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:17.952764 13125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 LEADER]: Becoming Leader. State: Replica: 7266775084784ff7809e5ac8a9ee8e1a, State: Running, Role: LEADER
I20260619 12:02:17.952916 13120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:17.952927 13125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:17.953343 13125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7266775084784ff7809e5ac8a9ee8e1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } } }
I20260619 12:02:17.954452 13120 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:02:17.954613 13125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:17.954757 13125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7266775084784ff7809e5ac8a9ee8e1a. Latest consensus state: current_term: 1 leader_uuid: "7266775084784ff7809e5ac8a9ee8e1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } } }
I20260619 12:02:17.954818 13125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:17.954954 13139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:17.957887 13139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:17.960557 13139 catalog_manager.cc:1347] Generated new cluster ID: 6ab24e9435554b2d995546456444f37a
I20260619 12:02:17.960623 13139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:02:17 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:17.966645 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:17.949812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.0.203.254:44931)
0619 12:02:17.949994 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:17.950003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:17.950034 (+ 31us) server_negotiation.cc:407] Connection header received
0619 12:02:17.950784 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:17.950794 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:17.950862 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:17.951015 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:17.951916 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:17.953435 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:17.954581 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:17.957281 (+ 2700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:17.957505 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:17.962188 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:17.962192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:17.962193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:17.962205 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:17.964395 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:17.964810 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:17.964813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:17.964813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:17.964924 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:17.965090 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:17.965091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:17.965091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:17.965542 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:17.965714 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:17.966184 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 12:02:17.966404 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260619 12:02:17.972733 13139 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:02:17.973932 13139 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:02:17.988732 13139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Generated new TSK 0
I20260619 12:02:17.989090 13139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:02:18.796898 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44931
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:18.886037 13165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:18.886272 13165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:18.886299 13165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:18.887547 13165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:18.887598 13165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:18.887629 13165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:02:18.889472 13165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.203.212:35441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:18.889734 13165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:18.889978 13165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:18.895876 13170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:18.895874 13173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:18.896147 13171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:18.896205 13165 server_base.cc:1034] running on GCE node
I20260619 12:02:18.896592 13165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:18.896929 13165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:18.898121 13165 hybrid_clock.cc:648] HybridClock initialized: now 1781870538898091 us; error 50 us; skew 500 ppm
Jun 19 12:02:18 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:18.901515 13165 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:02:18.902179 13165 webserver.cc:460] Webserver started at http://127.0.203.193:34941/ using document root <none> and password file <none>
I20260619 12:02:18.902488 13165 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:18.902565 13165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:18.902714 13165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:02:18 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:18.918357 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "6a7b2c35889b413ca37bf16dd1fdcb82"
format_stamp: "Formatted at 2026-06-19 12:02:18 on dist-test-slave-fbq8"
server_key: "c2c14063f1b6e40f76ea75a81f4e3bbd"
server_key_iv: "7e682e7646131b82b512c33d56fd9f2b"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:18.918960 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "6a7b2c35889b413ca37bf16dd1fdcb82"
format_stamp: "Formatted at 2026-06-19 12:02:18 on dist-test-slave-fbq8"
server_key: "c2c14063f1b6e40f76ea75a81f4e3bbd"
server_key_iv: "7e682e7646131b82b512c33d56fd9f2b"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:18.921084 13165 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:02:18.935448 13180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:18.935680 13165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:18.935773 13165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "6a7b2c35889b413ca37bf16dd1fdcb82"
format_stamp: "Formatted at 2026-06-19 12:02:18 on dist-test-slave-fbq8"
server_key: "c2c14063f1b6e40f76ea75a81f4e3bbd"
server_key_iv: "7e682e7646131b82b512c33d56fd9f2b"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:18.935876 13165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:18.964373 13165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:18.967164 13165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:18.967422 13165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:18.967764 13165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:18.968268 13165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:18.968364 13165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:18.968469 13165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:18.968541 13165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:18.975382 13165 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:38969
I20260619 12:02:18.975943 13165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:02:18.977124 13295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:38969 every 8 connection(s)
I20260619 12:02:18.982961 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13165
I20260619 12:02:18.983071 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:02:18.983278 815 external_mini_cluster.cc:1428] Setting key e8eb6a49db9cce255cc05f8235641197
Jun 19 12:02:18 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:18.992674 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:18.977848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44931 (local address 127.0.203.193:34677)
0619 12:02:18.978318 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:18.978371 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:18.979053 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:18.979571 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:18.979577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:18.979883 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:18.980192 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:18.980198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:18.981453 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:18.981455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:18.983989 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:18.983992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:18.984545 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:18.984546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:18.984760 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:18.985460 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:18.985474 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:18.987573 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:18.990312 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:18.990327 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:18.990331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:18.990760 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:18.991033 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:18.991034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:18.991034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:18.991397 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:18.991805 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:18.991815 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:18.992029 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 12:02:18.992138 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 12:02:18.992345 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":101,"threads_started":1}
I20260619 12:02:18.993672 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:18.978170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:34677 (local address 127.0.203.254:44931)
0619 12:02:18.979302 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:18.979304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:18.979312 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:02:18.979341 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:18.979342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:18.979355 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:18.979434 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:18.980503 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:18.981323 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:18.984085 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:18.984417 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:18.985029 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:18.987730 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:18.987731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:18.987732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:18.987741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:18.990160 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:18.990876 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:18.990877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:18.990878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:18.990953 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:18.991511 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:18.991512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:18.991512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:18.991701 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:18.993363 (+ 1662us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:18.993420 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 12:02:18.993534 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 12:02:18.994345 13296 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44931
I20260619 12:02:18.994469 13296 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:18.994715 13296 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:18.995779 13084 ts_manager.cc:194] Registered new tserver with Master: 6a7b2c35889b413ca37bf16dd1fdcb82 (127.0.203.193:38969)
I20260619 12:02:18.996471 13084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:34677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:02:19.025125 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44931
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 12:02:19.174825 13302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:19.175179 13302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:19.175261 13302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:19.177255 13302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:19.177507 13302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:19.177614 13302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:02:19.179854 13302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.203.212:35441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:19.180352 13302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:19.180706 13302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:19.187356 13310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:19.187783 13309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:19.188732 13302 server_base.cc:1034] running on GCE node
W20260619 12:02:19.188799 13312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:19.192829 13302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:19.193153 13302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:19.194308 13302 hybrid_clock.cc:648] HybridClock initialized: now 1781870539194289 us; error 37 us; skew 500 ppm
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:19.197939 13302 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:02:19.198487 13302 webserver.cc:460] Webserver started at http://127.0.203.194:41343/ using document root <none> and password file <none>
I20260619 12:02:19.198746 13302 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:19.198797 13302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:19.198920 13302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:19.220096 13302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "33d310366e194400af067ce02def3d50"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "d84c743543521d2c06c4bf78a15f50bf"
server_key_iv: "5cc8c23b52236484df034f3588f30ce5"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:19.220893 13302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "33d310366e194400af067ce02def3d50"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "d84c743543521d2c06c4bf78a15f50bf"
server_key_iv: "5cc8c23b52236484df034f3588f30ce5"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:19.228245 13302 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260619 12:02:19.253893 13319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:19.254169 13302 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 12:02:19.254266 13302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "33d310366e194400af067ce02def3d50"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "d84c743543521d2c06c4bf78a15f50bf"
server_key_iv: "5cc8c23b52236484df034f3588f30ce5"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:19.254367 13302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:19.267652 13302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:19.270499 13302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:19.270720 13302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:19.270967 13302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:19.271389 13302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:19.271469 13302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:19.271548 13302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:19.271610 13302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:19.278980 13302 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:43511
I20260619 12:02:19.279529 13302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:02:19.283795 13433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:43511 every 8 connection(s)
I20260619 12:02:19.288393 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13302
I20260619 12:02:19.288501 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:02:19.288864 815 external_mini_cluster.cc:1428] Setting key f2665e1f697837062cee95528b757a95
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:19.297550 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:19.283574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:52833 (local address 127.0.203.254:44931)
0619 12:02:19.283645 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:19.283648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:19.284363 (+ 715us) server_negotiation.cc:407] Connection header received
0619 12:02:19.285156 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:19.285158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:19.285177 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:19.285207 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:19.286056 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.286845 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.289264 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.289595 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.289639 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:19.294011 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:19.294013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:19.294014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:19.294022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:19.295840 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:19.296358 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:19.296360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:19.296360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:19.296416 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:19.296744 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:19.296745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:19.296746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:19.296907 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:19.296959 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:19.297325 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 12:02:19.297430 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 12:02:19.298434 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:19.283273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44931 (local address 127.0.203.194:52833)
0619 12:02:19.284223 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:19.284265 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:19.285026 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:19.285397 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:19.285401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:19.285664 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:19.285943 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:19.285949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.286963 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.286964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:19.289178 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:19.289181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.291363 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.291364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:19.291526 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:19.292168 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:19.292179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:19.293885 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:19.295950 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:19.295959 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:19.295962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:19.296272 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:19.296479 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:19.296480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:19.296480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:19.296670 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:19.296992 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:19.296998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:19.297172 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 12:02:19.298000 (+ 828us) client_negotiation.cc:240] Negotiation successful
0619 12:02:19.298185 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":93,"threads_started":1}
I20260619 12:02:19.299248 13434 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44931
I20260619 12:02:19.299345 13434 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:19.299566 13434 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:19.299937 13084 ts_manager.cc:194] Registered new tserver with Master: 33d310366e194400af067ce02def3d50 (127.0.203.194:43511)
I20260619 12:02:19.300369 13084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:52833
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:02:19.328552 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44931
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:19.452498 13439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:19.452975 13439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:19.453078 13439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:19.455214 13439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:19.455487 13439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:19.455595 13439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:02:19.458129 13439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.203.212:35441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:19.458671 13439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:19.459024 13439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:19.466641 13444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:19.467113 13439 server_base.cc:1034] running on GCE node
W20260619 12:02:19.467674 13445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:19.467962 13447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:19.468300 13439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:19.468633 13439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:19.469930 13439 hybrid_clock.cc:648] HybridClock initialized: now 1781870539469904 us; error 42 us; skew 500 ppm
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:19.474012 13439 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:02:19.474599 13439 webserver.cc:460] Webserver started at http://127.0.203.195:33845/ using document root <none> and password file <none>
I20260619 12:02:19.474866 13439 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:19.474922 13439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:19.475061 13439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:19.496053 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "860f8c6e498e495da300f15ca49aba24"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "08b67978904a6462aa7ea97762eda872"
server_key_iv: "ae56249c0282c9849b98b102a4bc0469"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:19.496532 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "860f8c6e498e495da300f15ca49aba24"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "08b67978904a6462aa7ea97762eda872"
server_key_iv: "ae56249c0282c9849b98b102a4bc0469"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:19.507261 13439 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.001s
I20260619 12:02:19.524847 13454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:19.525068 13439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260619 12:02:19.525163 13439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "860f8c6e498e495da300f15ca49aba24"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "08b67978904a6462aa7ea97762eda872"
server_key_iv: "ae56249c0282c9849b98b102a4bc0469"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:19.525277 13439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:19.552448 13439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:19.555087 13439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:19.555243 13439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:19.555500 13439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:19.555864 13439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:19.555907 13439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:19.555951 13439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:19.555975 13439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:19.563120 13439 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:41781
I20260619 12:02:19.563617 13439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:02:19.565966 13567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:41781 every 8 connection(s)
I20260619 12:02:19.573652 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13439
I20260619 12:02:19.573771 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:02:19.573968 815 external_mini_cluster.cc:1428] Setting key 229c5352ba604e488054835d48c78258
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:19.585811 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:19.567164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:36661 (local address 127.0.203.254:44931)
0619 12:02:19.567215 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:19.567217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:19.567806 (+ 589us) server_negotiation.cc:407] Connection header received
0619 12:02:19.568619 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:19.568621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:19.568663 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:19.568718 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:19.571152 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.571952 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.574500 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.574804 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.574877 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:19.581725 (+ 6848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:19.581728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:19.581729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:19.581739 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:19.583802 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:19.584445 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:19.584447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:19.584447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:19.584517 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:19.584918 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:19.584918 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:19.584919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:19.585083 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:19.585135 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:19.585594 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 12:02:19.585707 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 12:02:19.586433 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:19.566805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44931 (local address 127.0.203.195:36661)
0619 12:02:19.567560 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:19.567623 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:19.568473 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:19.570182 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:19.570187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:19.570411 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:19.570922 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:19.570930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.572726 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.572727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:19.574290 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:19.574297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:19.578267 (+ 3970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:19.578270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:19.578498 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:19.579285 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:19.579298 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:19.581525 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:19.583941 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:19.583953 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:19.583956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:19.584333 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:19.584612 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:19.584613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:19.584613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:19.584822 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:19.585191 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:19.585199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:19.585393 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 12:02:19.585972 (+ 579us) client_negotiation.cc:240] Negotiation successful
0619 12:02:19.586170 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":137,"threads_started":1}
I20260619 12:02:19.586926 13568 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44931
I20260619 12:02:19.587049 13568 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:19.587380 13568 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:19.587973 13084 ts_manager.cc:194] Registered new tserver with Master: 860f8c6e498e495da300f15ca49aba24 (127.0.203.195:41781)
I20260619 12:02:19.588613 13084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:36661
I20260619 12:02:19.596486 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:02:19.597469 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13165
I20260619 12:02:19.605268 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13302
I20260619 12:02:19.610976 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13439
I20260619 12:02:19.618351 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13052
I20260619 12:02:19.625679 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44931
--webserver_interface=127.0.203.254
--webserver_port=40863
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:44931
--ranger_config_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:19.709712 13571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:19.709920 13571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:19.709941 13571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:19.711159 13571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:19.711203 13571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:19.711217 13571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:19.711229 13571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:19.711242 13571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:19.712909 13571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:44931
--ranger_config_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:19.713173 13571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:19.713393 13571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:19.718569 13576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:19.718569 13577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:19.718582 13579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:19.719103 13571 server_base.cc:1034] running on GCE node
I20260619 12:02:19.719377 13571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:19.719731 13571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:19.720904 13571 hybrid_clock.cc:648] HybridClock initialized: now 1781870539720891 us; error 35 us; skew 500 ppm
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:19.723961 13571 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:19.724599 13571 webserver.cc:460] Webserver started at http://127.0.203.254:40863/ using document root <none> and password file <none>
I20260619 12:02:19.724895 13571 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:19.724949 13571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:19.799814 13571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:19.800818 13587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:19.801164 13571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:19.801272 13571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "7266775084784ff7809e5ac8a9ee8e1a"
format_stamp: "Formatted at 2026-06-19 12:02:17 on dist-test-slave-fbq8"
server_key: "d971e9afd36138385a913b89c916dcf4"
server_key_iv: "b71ebff5536d295e98aa52b1b87b2a04"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:19.801766 13571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:19.821719 13571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:19.824512 13571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:19.824723 13571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:19.829927 13571 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:44931
I20260619 12:02:19.829993 13639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:44931 every 8 connection(s)
I20260619 12:02:19.830406 13571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:19.831851 13640 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:02:19.832314 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13571
I20260619 12:02:19.832625 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:38969
--local_ip_for_outbound_sockets=127.0.203.193
--tserver_master_addrs=127.0.203.254:44931
--webserver_port=34941
--webserver_interface=127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:02:19.832787 13640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Bootstrap starting.
I20260619 12:02:19.841507 13640 log.cc:824] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:19.843350 13640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:02:19.843703 13640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Bootstrap complete.
I20260619 12:02:19.846256 13640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:19.846531 13640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266775084784ff7809e5ac8a9ee8e1a, State: Initialized, Role: FOLLOWER
I20260619 12:02:19.846709 13640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:19.846779 13640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:02:19.846827 13640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:02:19.846863 13640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:02:19.847920 13640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:19.848060 13640 leader_election.cc:302] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7266775084784ff7809e5ac8a9ee8e1a; no voters:
I20260619 12:02:19.848260 13640 leader_election.cc:288] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:02:19.848368 13644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:02:19.848523 13644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [term 2 LEADER]: Becoming Leader. State: Replica: 7266775084784ff7809e5ac8a9ee8e1a, State: Running, Role: LEADER
I20260619 12:02:19.848546 13640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:19.848755 13644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } }
I20260619 12:02:19.849105 13644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7266775084784ff7809e5ac8a9ee8e1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } } }
I20260619 12:02:19.849268 13644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:19.849442 13644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7266775084784ff7809e5ac8a9ee8e1a. Latest consensus state: current_term: 2 leader_uuid: "7266775084784ff7809e5ac8a9ee8e1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7266775084784ff7809e5ac8a9ee8e1a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44931 } } }
I20260619 12:02:19.849546 13644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:19.849737 13653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:19.850150 13640 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:02:19.850315 13653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:19.850498 13653 catalog_manager.cc:1259] Loaded cluster ID: 6ab24e9435554b2d995546456444f37a
I20260619 12:02:19.850567 13653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:02:19.851349 13653 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:02:19.851548 13653 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Loaded TSK: 0
I20260619 12:02:19.851773 13653 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:02:19.953946 13642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:19.954238 13642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:19.954277 13642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:19.956161 13642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:19.956224 13642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:19.956265 13642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:02:19.958535 13642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:38969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=34941
--tserver_master_addrs=127.0.203.254:44931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:19.958801 13642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:19.959069 13642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:19.968742 13677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:19.969163 13678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:19.969475 13680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:19.971750 13642 server_base.cc:1034] running on GCE node
I20260619 12:02:19.972113 13642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:19.972594 13642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:19.973790 13642 hybrid_clock.cc:648] HybridClock initialized: now 1781870539973697 us; error 125 us; skew 500 ppm
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:19.984717 13642 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:02:19.985396 13642 webserver.cc:460] Webserver started at http://127.0.203.193:34941/ using document root <none> and password file <none>
I20260619 12:02:19.985685 13642 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:19.985781 13642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:02:19 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:20.017733 13642 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260619 12:02:20.019930 13688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.020181 13642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:02:20.020367 13642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "6a7b2c35889b413ca37bf16dd1fdcb82"
format_stamp: "Formatted at 2026-06-19 12:02:18 on dist-test-slave-fbq8"
server_key: "c2c14063f1b6e40f76ea75a81f4e3bbd"
server_key_iv: "7e682e7646131b82b512c33d56fd9f2b"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:20.021272 13642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:20.052783 13642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:20.055413 13642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:20.055590 13642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:20.055858 13642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:20.056274 13642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:20.056327 13642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.056373 13642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:20.056399 13642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.064967 13642 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:38969
I20260619 12:02:20.065488 13642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:02:20.066560 13801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:38969 every 8 connection(s)
I20260619 12:02:20.070844 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13642
I20260619 12:02:20.071168 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:43511
--local_ip_for_outbound_sockets=127.0.203.194
--tserver_master_addrs=127.0.203.254:44931
--webserver_port=41343
--webserver_interface=127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:02:20 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:20.083820 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:20.067327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44931 (local address 127.0.203.193:41957)
0619 12:02:20.067612 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:20.067668 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:20.068329 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:20.069088 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:20.069093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:20.069467 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:20.069892 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:20.069899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.074011 (+ 4112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.074013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:20.075345 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:20.075348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.076703 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.076705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:20.076868 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:20.077521 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:20.077533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:20.079362 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:20.081791 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:20.081799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:20.081801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:20.082130 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:20.082435 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:20.082436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:20.082436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:20.082584 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:20.083101 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:20.083107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:20.083263 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 12:02:20.083374 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 12:02:20.083566 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260619 12:02:20.084452 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:20.067423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:41957 (local address 127.0.203.254:44931)
0619 12:02:20.067643 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:20.067653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:20.067761 (+ 108us) server_negotiation.cc:407] Connection header received
0619 12:02:20.068714 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:20.068722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:20.068791 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:20.068937 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:20.070078 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.073833 (+ 3755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.075469 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.076409 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.076551 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:20.079517 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:20.079523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:20.079523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:20.079550 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:20.081681 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.082236 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.082240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.082240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.082359 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.082659 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.082660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.082661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.082985 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:20.083060 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:20.084128 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0619 12:02:20.084258 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
I20260619 12:02:20.085143 13802 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44931
I20260619 12:02:20.085251 13802 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.085457 13802 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
W20260619 12:02:20.086319 13802 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:44931 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:02:20.086390 13802 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.086457 13802 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:20.086769 13802 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.086833 13802 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
W20260619 12:02:20.087082 13802 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:02:20.192030 13806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:20.192346 13806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:20.192391 13806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:20.194270 13806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:20.194350 13806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:20.194403 13806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:02:20.197068 13806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:43511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=41343
--tserver_master_addrs=127.0.203.254:44931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:20.197355 13806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:20.197656 13806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:20.203918 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:20.204149 13811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:20.203918 13812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:20.204730 13806 server_base.cc:1034] running on GCE node
I20260619 12:02:20.204979 13806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:20.205298 13806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:20.206524 13806 hybrid_clock.cc:648] HybridClock initialized: now 1781870540206490 us; error 52 us; skew 500 ppm
Jun 19 12:02:20 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:20.209416 13806 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:02:20.209910 13806 webserver.cc:460] Webserver started at http://127.0.203.194:41343/ using document root <none> and password file <none>
I20260619 12:02:20.210160 13806 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:20.210245 13806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:02:20 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:20.238499 13806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:20.239567 13822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.239786 13806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:20.239897 13806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "33d310366e194400af067ce02def3d50"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "d84c743543521d2c06c4bf78a15f50bf"
server_key_iv: "5cc8c23b52236484df034f3588f30ce5"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:20.240381 13806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:20.279115 13806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:20.281746 13806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:20.281920 13806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:20.282226 13806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:20.282629 13806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:20.282684 13806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.282739 13806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:20.282770 13806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.290058 13806 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:43511
I20260619 12:02:20.290594 13806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:02:20.291421 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13806
I20260619 12:02:20.291772 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.203.212:35441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:41781
--local_ip_for_outbound_sockets=127.0.203.195
--tserver_master_addrs=127.0.203.254:44931
--webserver_port=33845
--webserver_interface=127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:02:20.291790 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:43511 every 8 connection(s)
Jun 19 12:02:20 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:20.308023 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:20.292812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:44589 (local address 127.0.203.254:44931)
0619 12:02:20.292872 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:20.292874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:20.297023 (+ 4149us) server_negotiation.cc:407] Connection header received
0619 12:02:20.297752 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:20.297754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:20.297774 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:20.297859 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:20.298751 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.299582 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.301191 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.301559 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.301612 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:20.304365 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:20.304366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:20.304367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:20.304376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:20.306273 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.306834 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.306835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.306835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.306903 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.307201 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.307201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.307202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.307344 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:20.307394 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:20.307798 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 12:02:20.307909 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 12:02:20.308415 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:20.292467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44931 (local address 127.0.203.194:44589)
0619 12:02:20.296827 (+ 4360us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:20.296887 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:20.297598 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:20.298012 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:20.298017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:20.298302 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:20.298609 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:20.298616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.299725 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.299726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:20.301065 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:20.301069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.301724 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.301725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:20.301884 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:20.302582 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:20.302594 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:20.304226 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:20.306382 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:20.306389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:20.306392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:20.306728 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:20.306982 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:20.306983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:20.306983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:20.307120 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:20.307447 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:20.307455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:20.307642 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 12:02:20.307775 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 12:02:20.307934 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4302,"thread_start_us":112,"threads_started":1}
I20260619 12:02:20.308997 13936 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44931
I20260619 12:02:20.309126 13936 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.309330 13936 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
W20260619 12:02:20.310047 13936 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:44931 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:02:20.310138 13936 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.310223 13936 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:20.310546 13936 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.310633 13936 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
W20260619 12:02:20.310928 13936 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:02:20.423224 13939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:20.423455 13939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:20.423486 13939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:20.424981 13939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:20.425052 13939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:20.425107 13939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:02:20.426890 13939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.203.212:35441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:41781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=33845
--tserver_master_addrs=127.0.203.254:44931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:20.427196 13939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:20.427460 13939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:20.433710 13945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:20.433708 13948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:20.434171 13939 server_base.cc:1034] running on GCE node
W20260619 12:02:20.434432 13946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:20.434737 13939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:20.435047 13939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:20.436210 13939 hybrid_clock.cc:648] HybridClock initialized: now 1781870540436198 us; error 30 us; skew 500 ppm
Jun 19 12:02:20 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:20.439208 13939 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:02:20.439764 13939 webserver.cc:460] Webserver started at http://127.0.203.195:33845/ using document root <none> and password file <none>
I20260619 12:02:20.440052 13939 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:20.440133 13939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:02:20 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:20.462385 13939 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:20.463243 13956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.463460 13939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:20.463572 13939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "860f8c6e498e495da300f15ca49aba24"
format_stamp: "Formatted at 2026-06-19 12:02:19 on dist-test-slave-fbq8"
server_key: "08b67978904a6462aa7ea97762eda872"
server_key_iv: "ae56249c0282c9849b98b102a4bc0469"
server_key_version: "kuduclusterkey@0"
I20260619 12:02:20.464048 13939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:20.482940 13939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:20.485626 13939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:20.485795 13939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:20.486068 13939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:20.486487 13939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:20.486542 13939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.486577 13939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:20.486600 13939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:20.494385 13939 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:41781
I20260619 12:02:20.494464 14069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:41781 every 8 connection(s)
I20260619 12:02:20.494886 13939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:02:20 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:20.502822 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 13939
I20260619 12:02:20.506873 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:20.495810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:51443 (local address 127.0.203.254:44931)
0619 12:02:20.495892 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:20.495894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:20.496050 (+ 156us) server_negotiation.cc:407] Connection header received
0619 12:02:20.496537 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:20.496538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:20.496551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:20.496634 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:20.497459 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.498217 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.499385 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.499679 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.499772 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:20.502308 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:20.502309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:20.502310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:20.502318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:20.504160 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.505674 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.505675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.505676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.505745 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.506060 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.506060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.506061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.506193 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:20.506261 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:20.506671 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 12:02:20.506763 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:02:20.507347 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:20.495752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44931 (local address 127.0.203.195:51443)
0619 12:02:20.495912 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:20.495953 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:20.496435 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:20.496850 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:20.496854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:20.497121 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:20.497341 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:20.497345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.498324 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.498325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:20.499285 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:20.499288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.499809 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.499810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:20.499919 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:20.500584 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:20.500592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:20.502203 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:20.504262 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:20.504270 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:20.504273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:20.505589 (+ 1316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:20.505831 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:20.505832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:20.505833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:20.505981 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:20.506298 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:20.506305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:20.506483 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 12:02:20.506629 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 12:02:20.506805 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260619 12:02:20.507934 14070 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44931
I20260619 12:02:20.508049 14070 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.508270 14070 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
W20260619 12:02:20.508952 14070 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:44931 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:02:20.509040 14070 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.509327 14070 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:20.509873 14070 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:20.509964 14070 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
W20260619 12:02:20.510249 14070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:02:20.511590 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:20.503264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.0.203.254:44931)
0619 12:02:20.503408 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:20.503410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:20.503718 (+ 308us) server_negotiation.cc:407] Connection header received
0619 12:02:20.503797 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:20.503798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:20.503820 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:20.503900 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:20.504677 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.505475 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.506378 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:20.506660 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:20.506721 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:20.508839 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:20.508840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:20.508841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:20.508849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:20.510440 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.510776 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.510777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.510778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.510843 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:20.511017 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:20.511018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:20.511019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:20.511154 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:20.511343 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:20.511400 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 12:02:20.511489 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
Jun 19 12:02:21 dist-test-slave-fbq8 krb5kdc[12608](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781870541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:21.087317 13802 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:21.087508 13802 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:21.088032 13604 ts_manager.cc:194] Registered new tserver with Master: 6a7b2c35889b413ca37bf16dd1fdcb82 (127.0.203.193:38969)
I20260619 12:02:21.089082 13604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:41957
I20260619 12:02:21.311182 13936 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:21.311365 13936 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:21.311873 13604 ts_manager.cc:194] Registered new tserver with Master: 33d310366e194400af067ce02def3d50 (127.0.203.194:43511)
I20260619 12:02:21.312419 13604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:44589
Jun 19 12:02:21 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:21.510509 14070 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:21.510728 14070 heartbeater.cc:511] Master 127.0.203.254:44931 requested a full tablet report, sending...
I20260619 12:02:21.511096 13604 ts_manager.cc:194] Registered new tserver with Master: 860f8c6e498e495da300f15ca49aba24 (127.0.203.195:41781)
I20260619 12:02:21.511868 13604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:51443
I20260619 12:02:21.517480 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:02:21.528077 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.518532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.0.203.254:44931)
0619 12:02:21.518696 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:21.518699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:21.518856 (+ 157us) server_negotiation.cc:407] Connection header received
0619 12:02:21.518935 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:21.518936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:21.518951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:21.519042 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:21.519785 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.520597 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.521749 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.522078 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.522152 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.524075 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:21.524076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:21.524077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:21.524086 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:21.526251 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:21.526787 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:21.526788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:21.526788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:21.526895 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:21.527162 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:21.527162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:21.527163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:21.527302 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:21.527394 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:21.527871 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 12:02:21.527959 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260619 12:02:21.531450 13604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:02:21.534611 13604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:02:21.552568 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.547112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.0.203.193:38969)
0619 12:02:21.547414 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:21.547420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:21.547834 (+ 414us) server_negotiation.cc:407] Connection header received
0619 12:02:21.547938 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:21.547940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:21.547956 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:21.548006 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:21.548286 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.549627 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.551266 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.552044 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.552093 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.552159 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:21.552430 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 12:02:21.552473 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":73,"threads_started":1}
I20260619 12:02:21.553084 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.547000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:38969 (local address 127.0.0.1:53874)
0619 12:02:21.547777 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:21.547786 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:21.547856 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:21.548123 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:21.548126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:21.548128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:21.548206 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.548209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.549731 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.549732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.551159 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.551162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.552222 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.552223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.552312 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.552325 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 12:02:21.552924 (+ 599us) client_negotiation.cc:240] Negotiation successful
0619 12:02:21.553003 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":131,"threads_started":1}
I20260619 12:02:21.553691 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.546899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:41781 (local address 127.0.0.1:46746)
0619 12:02:21.547520 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:21.547574 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:21.547665 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:21.548894 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:21.548895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:21.548898 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:21.548966 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.548969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.550714 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.550715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.551874 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.551877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.553400 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.553402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.553480 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.553485 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:02:21.553542 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 12:02:21.553595 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":124,"threads_started":1}
I20260619 12:02:21.553982 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.547238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.0.203.195:41781)
0619 12:02:21.548504 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:21.548510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:21.548523 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:02:21.548567 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:21.548569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:21.548588 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:21.548760 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:21.549064 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.550603 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.551979 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.553167 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.553249 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.553354 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:21.553840 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 12:02:21.553892 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":67,"threads_started":1}
I20260619 12:02:21.554095 13736 tablet_service.cc:1467] Processing CreateTablet for tablet b2dd2a087d764a88a1b334f894becea8 (DEFAULT_TABLE table=test-table [id=3834202155184322b61bb97b9d67e57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:21.554509 13736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2dd2a087d764a88a1b334f894becea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:21.554694 13992 tablet_service.cc:1467] Processing CreateTablet for tablet b2dd2a087d764a88a1b334f894becea8 (DEFAULT_TABLE table=test-table [id=3834202155184322b61bb97b9d67e57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:21.555034 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2dd2a087d764a88a1b334f894becea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:21.558965 14104 tablet_bootstrap.cc:492] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Bootstrap starting.
I20260619 12:02:21.559024 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.547265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:43511 (local address 127.0.0.1:43092)
0619 12:02:21.548284 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:21.548298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:21.548364 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:21.554416 (+ 6052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:21.554418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:21.554421 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:21.554474 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.554476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.556216 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.556217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.557429 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.557432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.558783 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.558784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.558850 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.558856 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:02:21.558888 (+ 32us) client_negotiation.cc:240] Negotiation successful
0619 12:02:21.558935 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":104,"threads_started":1}
I20260619 12:02:21.559427 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.547716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.0.203.194:43511)
0619 12:02:21.554184 (+ 6468us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:21.554190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:21.554204 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:02:21.554250 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:21.554253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:21.554287 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:21.554558 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:21.554663 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.556108 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.557555 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.558676 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.559161 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.559258 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:21.559300 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 12:02:21.559346 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6415,"thread_start_us":78,"threads_started":1}
I20260619 12:02:21.560005 14104 tablet_bootstrap.cc:654] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:21.560186 13870 tablet_service.cc:1467] Processing CreateTablet for tablet b2dd2a087d764a88a1b334f894becea8 (DEFAULT_TABLE table=test-table [id=3834202155184322b61bb97b9d67e57c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:21.560371 14104 log.cc:824] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:21.560700 13870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2dd2a087d764a88a1b334f894becea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:21.561396 14104 tablet_bootstrap.cc:492] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: No bootstrap required, opened a new log
I20260619 12:02:21.561460 14104 ts_tablet_manager.cc:1397] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:02:21.564006 14104 raft_consensus.cc:348] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.564177 14104 raft_consensus.cc:374] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:21.564208 14104 raft_consensus.cc:729] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860f8c6e498e495da300f15ca49aba24, State: Initialized, Role: FOLLOWER
I20260619 12:02:21.564627 14104 consensus_queue.cc:260] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.565034 14104 ts_tablet_manager.cc:1428] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260619 12:02:21.565201 14070 heartbeater.cc:503] Master 127.0.203.254:44931 was elected leader, sending a full tablet report...
I20260619 12:02:21.565171 14103 tablet_bootstrap.cc:492] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82: Bootstrap starting.
I20260619 12:02:21.565487 14106 tablet_bootstrap.cc:492] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: Bootstrap starting.
I20260619 12:02:21.565877 14103 tablet_bootstrap.cc:654] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:21.566213 14106 tablet_bootstrap.cc:654] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:21.566218 14103 log.cc:824] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:21.566571 14106 log.cc:824] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:21.567422 14106 tablet_bootstrap.cc:492] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: No bootstrap required, opened a new log
I20260619 12:02:21.567516 14106 ts_tablet_manager.cc:1397] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:02:21.567691 14103 tablet_bootstrap.cc:492] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82: No bootstrap required, opened a new log
I20260619 12:02:21.567806 14103 ts_tablet_manager.cc:1397] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:02:21.570509 14106 raft_consensus.cc:348] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.570704 14106 raft_consensus.cc:374] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:21.570758 14106 raft_consensus.cc:729] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d310366e194400af067ce02def3d50, State: Initialized, Role: FOLLOWER
I20260619 12:02:21.570930 14106 consensus_queue.cc:260] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.571238 14106 ts_tablet_manager.cc:1428] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 12:02:21.571452 13936 heartbeater.cc:503] Master 127.0.203.254:44931 was elected leader, sending a full tablet report...
I20260619 12:02:21.571209 14103 raft_consensus.cc:348] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.571859 14103 raft_consensus.cc:374] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:21.571902 14103 raft_consensus.cc:729] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7b2c35889b413ca37bf16dd1fdcb82, State: Initialized, Role: FOLLOWER
I20260619 12:02:21.572016 14103 consensus_queue.cc:260] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.572284 14103 ts_tablet_manager.cc:1428] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 12:02:21.572398 13802 heartbeater.cc:503] Master 127.0.203.254:44931 was elected leader, sending a full tablet report...
Jun 19 12:02:21 dist-test-slave-fbq8 krb5kdc[12608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for HTTP/127.0.203.212@KRBTEST.COM
I20260619 12:02:21.720572 14111 raft_consensus.cc:482] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:21.720762 14111 raft_consensus.cc:504] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.721129 14111 leader_election.cc:288] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 860f8c6e498e495da300f15ca49aba24 (127.0.203.195:41781), 33d310366e194400af067ce02def3d50 (127.0.203.194:43511)
I20260619 12:02:21.725286 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.721483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:57597 (local address 127.0.203.195:41781)
0619 12:02:21.721548 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:21.721550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:21.721715 (+ 165us) server_negotiation.cc:407] Connection header received
0619 12:02:21.721817 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:21.721818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:21.721827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:21.721856 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:21.722088 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.722841 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.724266 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.724891 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.724942 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.724952 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:21.725163 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 12:02:21.725208 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 12:02:21.725584 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.721273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:41781 (local address 127.0.203.193:57597)
0619 12:02:21.721659 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:21.721671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:21.721741 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:21.721928 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:21.721929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:21.721930 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:21.722021 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.722023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.722948 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.722949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.724159 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.724162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.725014 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.725015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.725086 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.725090 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:02:21.725431 (+ 341us) client_negotiation.cc:240] Negotiation successful
0619 12:02:21.725481 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":84,"threads_started":1}
I20260619 12:02:21.733021 14008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd2a087d764a88a1b334f894becea8" candidate_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860f8c6e498e495da300f15ca49aba24" is_pre_election: true
I20260619 12:02:21.733268 14008 raft_consensus.cc:2454] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7b2c35889b413ca37bf16dd1fdcb82 in term 0.
I20260619 12:02:21.733759 13689 leader_election.cc:302] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7b2c35889b413ca37bf16dd1fdcb82, 860f8c6e498e495da300f15ca49aba24; no voters:
I20260619 12:02:21.733922 14111 raft_consensus.cc:2781] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:02:21.733981 14111 raft_consensus.cc:482] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:02:21.734014 14111 raft_consensus.cc:3037] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:21.734977 14111 raft_consensus.cc:504] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.735144 14111 leader_election.cc:288] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [CANDIDATE]: Term 1 election: Requested vote from peers 860f8c6e498e495da300f15ca49aba24 (127.0.203.195:41781), 33d310366e194400af067ce02def3d50 (127.0.203.194:43511)
I20260619 12:02:21.735580 14008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd2a087d764a88a1b334f894becea8" candidate_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860f8c6e498e495da300f15ca49aba24"
I20260619 12:02:21.735675 14008 raft_consensus.cc:3037] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:21.737154 14008 raft_consensus.cc:2454] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7b2c35889b413ca37bf16dd1fdcb82 in term 1.
I20260619 12:02:21.737352 13689 leader_election.cc:302] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7b2c35889b413ca37bf16dd1fdcb82, 860f8c6e498e495da300f15ca49aba24; no voters:
I20260619 12:02:21.737483 14111 raft_consensus.cc:2781] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:21.737673 14111 raft_consensus.cc:686] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7b2c35889b413ca37bf16dd1fdcb82, State: Running, Role: LEADER
I20260619 12:02:21.737789 14111 consensus_queue.cc:237] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } }
I20260619 12:02:21.738680 13603 catalog_manager.cc:5495] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7b2c35889b413ca37bf16dd1fdcb82 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 38969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 } health_report { overall_health: UNKNOWN } } }
W20260619 12:02:21.746279 14071 tablet.cc:2369] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:21.746917 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.721700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:56579 (local address 127.0.203.194:43511)
0619 12:02:21.721747 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:21.721749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:21.723158 (+ 1409us) server_negotiation.cc:407] Connection header received
0619 12:02:21.742407 (+ 19249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:21.742409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:21.742423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:21.742456 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:21.743109 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.743851 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.745202 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.745762 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.745818 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.745830 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:21.746788 (+ 958us) server_negotiation.cc:299] Negotiation successful
0619 12:02:21.746840 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:02:21.747466 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.721487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:43511 (local address 127.0.203.193:56579)
0619 12:02:21.723081 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:21.723095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:21.742277 (+ 19182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:21.742938 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:21.742939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:21.742942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:21.743022 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.743024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.743951 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.743952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.745098 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:21.745101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.746604 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.746605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:21.746685 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.746690 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:02:21.747316 (+ 626us) client_negotiation.cc:240] Negotiation successful
0619 12:02:21.747376 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"thread_start_us":104,"threads_started":1}
I20260619 12:02:21.747831 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd2a087d764a88a1b334f894becea8" candidate_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d310366e194400af067ce02def3d50"
I20260619 12:02:21.747828 13890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd2a087d764a88a1b334f894becea8" candidate_uuid: "6a7b2c35889b413ca37bf16dd1fdcb82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d310366e194400af067ce02def3d50" is_pre_election: true
I20260619 12:02:21.748036 13890 raft_consensus.cc:2454] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7b2c35889b413ca37bf16dd1fdcb82 in term 0.
I20260619 12:02:21.783816 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.780555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.0.203.193:38969)
0619 12:02:21.780599 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:21.780600 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:21.780608 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:02:21.780670 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:21.780671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:21.780693 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:21.780770 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:02:21.780943 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.781693 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.782688 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:21.782985 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:21.783041 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:21.783264 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:02:21.783406 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:02:21.783479 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:21.783710 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 12:02:21.783734 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 12:02:21.786350 13890 raft_consensus.cc:3037] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:21.787369 13890 raft_consensus.cc:1261] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 1 FOLLOWER]: Refusing update from remote peer 6a7b2c35889b413ca37bf16dd1fdcb82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:21.787744 14111 consensus_queue.cc:1035] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d310366e194400af067ce02def3d50" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:21.793862 14008 raft_consensus.cc:1261] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 1 FOLLOWER]: Refusing update from remote peer 6a7b2c35889b413ca37bf16dd1fdcb82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 12:02:21.794149 13937 tablet.cc:2369] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:21.794548 14111 consensus_queue.cc:1035] T b2dd2a087d764a88a1b334f894becea8 P 6a7b2c35889b413ca37bf16dd1fdcb82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "860f8c6e498e495da300f15ca49aba24" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:21.798698 14119 mvcc.cc:205] Tried to move back new op lower bound from 7298541739154685952 to 7298541738958680064. Current Snapshot: MvccSnapshot[applied={T|T < 7298541739154685952}]
I20260619 12:02:21.798916 14122 mvcc.cc:205] Tried to move back new op lower bound from 7298541739154685952 to 7298541738958680064. Current Snapshot: MvccSnapshot[applied={T|T < 7298541739154685952}]
I20260619 12:02:21.801932 14120 mvcc.cc:205] Tried to move back new op lower bound from 7298541739154685952 to 7298541738958680064. Current Snapshot: MvccSnapshot[applied={T|T < 7298541739154685952}]
I20260619 12:02:21.804739 13603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:21.804836 13603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:21.805694 13603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7266775084784ff7809e5ac8a9ee8e1a: Sending DeleteTablet for 3 replicas of tablet b2dd2a087d764a88a1b334f894becea8
I20260619 12:02:21.806075 13870 tablet_service.cc:1514] Processing DeleteTablet for tablet b2dd2a087d764a88a1b334f894becea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:21 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:43092
I20260619 12:02:21.806403 13992 tablet_service.cc:1514] Processing DeleteTablet for tablet b2dd2a087d764a88a1b334f894becea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:21 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:46746
I20260619 12:02:21.806895 13736 tablet_service.cc:1514] Processing DeleteTablet for tablet b2dd2a087d764a88a1b334f894becea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:21 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:53874
I20260619 12:02:21.807281 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13642
I20260619 12:02:21.809049 14129 tablet_replica.cc:330] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: stopping tablet replica
I20260619 12:02:21.809213 14129 raft_consensus.cc:2229] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:21.809396 14129 raft_consensus.cc:2258] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:21.809888 14129 ts_tablet_manager.cc:1905] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:21.811585 14129 ts_tablet_manager.cc:1918] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:21.811731 14129 log.cc:1196] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/b2dd2a087d764a88a1b334f894becea8
I20260619 12:02:21.812071 14129 ts_tablet_manager.cc:1939] T b2dd2a087d764a88a1b334f894becea8 P 860f8c6e498e495da300f15ca49aba24: Deleting consensus metadata
I20260619 12:02:21.812601 13588 catalog_manager.cc:4841] TS 860f8c6e498e495da300f15ca49aba24 (127.0.203.195:41781): tablet b2dd2a087d764a88a1b334f894becea8 (table test-table [id=3834202155184322b61bb97b9d67e57c]) successfully deleted
I20260619 12:02:21.816753 14128 tablet_replica.cc:330] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50: stopping tablet replica
I20260619 12:02:21.816933 14128 raft_consensus.cc:2229] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:21.817092 14128 raft_consensus.cc:2258] T b2dd2a087d764a88a1b334f894becea8 P 33d310366e194400af067ce02def3d50 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 12:02:21.820407 13591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:38969 (error 108)
I20260619 12:02:21.820766 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13806
I20260619 12:02:21.822502 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:21.822343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:38969 (local address 127.0.0.1:53898)
0619 12:02:21.822399 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:21.822461 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:38969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 12:02:21.827510 13591 catalog_manager.cc:4584] TS 6a7b2c35889b413ca37bf16dd1fdcb82 (127.0.203.193:38969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2dd2a087d764a88a1b334f894becea8: Network error: Client connection negotiation failed: client connection to 127.0.203.193:38969: connect: Connection refused (error 111)
I20260619 12:02:21.828085 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13939
I20260619 12:02:21.836933 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 13571
I20260619 12:02:21.843761 815 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 12:02:22.188808 815 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 12:02:22.188953 815 mini_ranger.cc:66] Stopping Ranger...
I20260619 12:02:22.533977 815 mini_ranger.cc:68] Stopped Ranger
2026-06-19 12:02:22.534 UTC [12638] LOG: received smart shutdown request
2026-06-19 12:02:22.537 UTC [12638] LOG: background worker "logical replication launcher" (PID 12647) exited with exit code 1
2026-06-19 12:02:22.538 UTC [12642] LOG: shutting down
2026-06-19 12:02:22.741 UTC [12638] LOG: database system is shut down
2026-06-19T12:02:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52180 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:02:22 dist-test-slave-fbq8 krb5kdc[14140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:02:22 dist-test-slave-fbq8 krb5kdc[14140](info): set up 2 sockets
Jun 19 12:02:22 dist-test-slave-fbq8 krb5kdc[14140](info): commencing operation
krb5kdc: starting...
W20260619 12:02:23.957139 815 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:02:24.108286 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:45167
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:45167,127.0.203.253:44789,127.0.203.252:38329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:24.195138 14153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:24.195381 14153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:24.195407 14153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:24.196728 14153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:24.196797 14153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:24.196815 14153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:24.196827 14153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:24.196839 14153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:24.198494 14153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:45167,127.0.203.253:44789,127.0.203.252:38329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:24.198763 14153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:24.199034 14153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:24.204819 14161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:24.204852 14158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:24.204890 14159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:24.205214 14153 server_base.cc:1034] running on GCE node
I20260619 12:02:24.205585 14153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:24.205955 14153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:24.207167 14153 hybrid_clock.cc:648] HybridClock initialized: now 1781870544207141 us; error 49 us; skew 500 ppm
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:24.210631 14153 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:24.211210 14153 webserver.cc:460] Webserver started at http://127.0.203.254:33191/ using document root <none> and password file <none>
I20260619 12:02:24.211457 14153 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:24.211535 14153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:24.211709 14153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:24.212950 14153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "9a3ce1496c0f46509f8022423d95f91a"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "4427ea845d2eee3ccdead726ccfa6371"
server_key_iv: "fb0e50056c5cc98830ee8d3f3fedbc4c"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.213335 14153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "9a3ce1496c0f46509f8022423d95f91a"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "4427ea845d2eee3ccdead726ccfa6371"
server_key_iv: "fb0e50056c5cc98830ee8d3f3fedbc4c"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.214970 14153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:24.215988 14168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:24.216279 14153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:24.216418 14153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "9a3ce1496c0f46509f8022423d95f91a"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "4427ea845d2eee3ccdead726ccfa6371"
server_key_iv: "fb0e50056c5cc98830ee8d3f3fedbc4c"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.216500 14153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:24.235385 14153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:24.238169 14153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:24.238381 14153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:24.243674 14153 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:45167
I20260619 12:02:24.243736 14220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:45167 every 8 connection(s)
I20260619 12:02:24.244175 14153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:24.244838 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14153
I20260619 12:02:24.244969 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:02:24.245226 815 external_mini_cluster.cc:1428] Setting key 6e0dc0ae7704c416e7c0fd0ce6d0495b
I20260619 12:02:24.245590 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:24.247660 14221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.253@KRBTEST.COM" created.
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:24.263334 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.248687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:45167 (local address 127.0.0.1:54354)
0619 12:02:24.248944 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.248994 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.249651 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.249991 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.249994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.250331 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.250530 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.250535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.251830 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.251831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.252611 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.252613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.253278 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.253279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.253356 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.254057 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.254071 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.258171 (+ 4100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.261377 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.261384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.261387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.261610 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.261871 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.261872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.261872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.261959 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.262344 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.262349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.262418 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.262516 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.262768 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":87,"thread_start_us":111,"threads_started":1}
I20260619 12:02:24.263334 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.248965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.0.203.254:45167)
0619 12:02:24.249146 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.249152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.249188 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:02:24.249829 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.249832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.249854 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.249983 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.250630 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.251665 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.252771 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.253171 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.253315 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.258327 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.258341 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.258341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.258353 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.261262 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.261706 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.261715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.261716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.261804 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.262027 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.262028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.262028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.262278 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.262369 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.262562 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.262702 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260619 12:02:24.264389 14221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:02:24.264935 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.264784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:46282)
0619 12:02:24.264842 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.264900 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 12:02:24.265347 14170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
I20260619 12:02:24.265934 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.265829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:46294)
0619 12:02:24.265873 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.265906 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 12:02:24.266084 14221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:44789: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab.
Entry for principal kudu/127.0.203.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab.
I20260619 12:02:24.276858 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:44789
--webserver_interface=127.0.203.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab
--principal=kudu/127.0.203.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:45167,127.0.203.253:44789,127.0.203.252:38329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:02:24.315382 14221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } attempt: 1
I20260619 12:02:24.316095 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.315918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:43362)
0619 12:02:24.315997 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.316047 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 12:02:24.316689 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.316540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:43378)
0619 12:02:24.316599 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.316626 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 12:02:24.316816 14221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:44789: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
W20260619 12:02:24.363657 14228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:24.363876 14228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:24.363900 14228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:24.365106 14228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:24.365160 14228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:24.365175 14228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:24.365190 14228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:24.365203 14228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:24.366776 14228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:45167,127.0.203.253:44789,127.0.203.252:38329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab
--principal=kudu/127.0.203.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:24.367048 14228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:24.367305 14228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:24.372779 14233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:24.372875 14236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:24.372920 14228 server_base.cc:1034] running on GCE node
W20260619 12:02:24.372920 14234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:24.373375 14228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:24.373770 14228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:24.374954 14228 hybrid_clock.cc:648] HybridClock initialized: now 1781870544374931 us; error 51 us; skew 500 ppm
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:24.378149 14228 init.cc:377] Logged in from keytab as kudu/127.0.203.253@KRBTEST.COM (short username kudu)
I20260619 12:02:24.378682 14228 webserver.cc:460] Webserver started at http://127.0.203.253:36527/ using document root <none> and password file <none>
I20260619 12:02:24.378961 14228 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:24.379030 14228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:24.379194 14228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:24.380461 14228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data/instance:
uuid: "42a3be83de1e49bdb13124f5fed0cfdb"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "e838fef6de6f2b034e0a88acfe34d0fd"
server_key_iv: "f52e051c4c47670ecba4e0e410cdabcc"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.380875 14228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/wal/instance:
uuid: "42a3be83de1e49bdb13124f5fed0cfdb"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "e838fef6de6f2b034e0a88acfe34d0fd"
server_key_iv: "f52e051c4c47670ecba4e0e410cdabcc"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.382573 14228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:24.383533 14243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:24.383813 14228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:24.383924 14228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/wal
uuid: "42a3be83de1e49bdb13124f5fed0cfdb"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "e838fef6de6f2b034e0a88acfe34d0fd"
server_key_iv: "f52e051c4c47670ecba4e0e410cdabcc"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.384003 14228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:24.385006 14221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } attempt: 2
I20260619 12:02:24.385733 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.385544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:43388)
0619 12:02:24.385642 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.385699 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 12:02:24.386286 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.386105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:43396)
0619 12:02:24.386230 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.386265 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260619 12:02:24.386430 14221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:44789: Network error: Client connection negotiation failed: client connection to 127.0.203.253:44789: connect: Connection refused (error 111)
I20260619 12:02:24.410802 14228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:24.413820 14228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:24.414005 14228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:24.419373 14228 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.253:44789
I20260619 12:02:24.419399 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.253:44789 every 8 connection(s)
I20260619 12:02:24.420185 14228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
I20260619 12:02:24.421408 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:24.423187 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14228
I20260619 12:02:24.423310 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-1/wal/instance
I20260619 12:02:24.423509 815 external_mini_cluster.cc:1428] Setting key c212d4dcf44501296420a286d41efad7
I20260619 12:02:24.423609 14296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.252@KRBTEST.COM; defaulting to no policy
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.252@KRBTEST.COM" created.
I20260619 12:02:24.434485 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.424757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.0.203.254:45167)
0619 12:02:24.424843 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.424846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.425085 (+ 239us) server_negotiation.cc:407] Connection header received
0619 12:02:24.425783 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.425784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.425804 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.425870 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.426616 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.427161 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.428259 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.428499 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.428580 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.431287 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.431288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.431289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.431299 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.433015 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.433448 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.433450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.433450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.433507 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.433748 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.433749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.433749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.433912 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.434004 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.434285 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.434378 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:02:24.434713 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.424694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:45167 (local address 127.0.0.1:49462)
0619 12:02:24.424922 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.424973 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.425643 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.425961 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.425965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.426273 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.426519 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.426524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.427258 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.427260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.428162 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.428164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.428585 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.428586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.428792 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.429608 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.429618 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.431188 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.433120 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.433127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.433130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.433364 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.433578 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.433579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.433579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.433684 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.433995 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.434002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.434149 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.434253 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.434448 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":104,"threads_started":1}
I20260619 12:02:24.435580 14296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:02:24.444108 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.435942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:43400)
0619 12:02:24.436180 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.436192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.436257 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.436401 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.436402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.436610 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.436724 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.436727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.437672 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.437673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.438269 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.438271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.438757 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.438758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.438827 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.439306 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.439311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.440790 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.442952 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.442953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.442954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.443128 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.443341 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.443342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.443343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.443399 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.443749 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.443751 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.443806 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.443881 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.444033 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260619 12:02:24.444314 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.436015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.203.253:44789)
0619 12:02:24.436189 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.436196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.436251 (+ 55us) server_negotiation.cc:407] Connection header received
0619 12:02:24.436318 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.436321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.436336 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.436402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.436826 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.437580 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.438362 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.438680 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.438760 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.440897 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.440904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.440905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.440911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.442829 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.443219 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.443223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.443223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.443279 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.443469 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.443470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.443471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.443649 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.443720 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.443973 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.444144 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260619 12:02:24.445238 14296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:02:24.445695 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.445518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35450)
0619 12:02:24.445596 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.445659 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 12:02:24.445921 14244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
I20260619 12:02:24.446728 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.446593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35454)
0619 12:02:24.446672 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.446706 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 12:02:24.446950 14296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:38329: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab.
Entry for principal kudu/127.0.203.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab.
I20260619 12:02:24.451957 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:38329
--webserver_interface=127.0.203.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab
--principal=kudu/127.0.203.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:45167,127.0.203.253:44789,127.0.203.252:38329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:02:24.477639 14221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } attempt: 3
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:02:24.484972 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.478179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.0.203.253:44789)
0619 12:02:24.478274 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.478276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.478285 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:02:24.478309 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.478310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.478319 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.478381 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.478728 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.479282 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.479967 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.480183 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.480259 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.482333 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.482335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.482335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.482343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.483787 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.484170 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.484171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.484172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.484226 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.484414 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.484415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.484415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.484528 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.484595 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.484789 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.484865 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:02:24.484972 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.478050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:43408)
0619 12:02:24.478117 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.478130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.478181 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.478386 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.478387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.478555 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.478642 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.478644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.479366 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.479367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.479890 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.479891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.480254 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.480255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.480367 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.480994 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.480999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.482236 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.483899 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.483900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.483901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.484080 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.484289 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.484290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.484290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.484353 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.484598 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.484599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.484688 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.484756 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.484865 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 12:02:24.485729 14221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:02:24.486100 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.486001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35466)
0619 12:02:24.486044 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.486073 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 12:02:24.486404 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.486325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35478)
0619 12:02:24.486360 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.486380 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 12:02:24.486492 14221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:38329: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
I20260619 12:02:24.496212 14296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } attempt: 1
I20260619 12:02:24.496798 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.496627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35486)
0619 12:02:24.496709 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.496771 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:02:24.497269 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.497169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35496)
0619 12:02:24.497220 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.497246 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 12:02:24.497372 14296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:38329: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
I20260619 12:02:24.517652 14221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } attempt: 1
I20260619 12:02:24.518245 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.518102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35510)
0619 12:02:24.518168 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.518213 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 12:02:24.518620 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.518522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35522)
0619 12:02:24.518568 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.518596 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 12:02:24.518723 14221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:38329: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
W20260619 12:02:24.545698 14303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:24.545950 14303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:24.545974 14303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:24.547158 14303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:24.547207 14303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:24.547221 14303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:24.547235 14303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:24.547246 14303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:24.548935 14303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:45167,127.0.203.253:44789,127.0.203.252:38329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab
--principal=kudu/127.0.203.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:38329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:24.549238 14303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:24.549520 14303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:24.555364 14311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:24.555419 14308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:24.555441 14309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:24.555769 14303 server_base.cc:1034] running on GCE node
I20260619 12:02:24.556048 14303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:24.556368 14303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:24.557559 14303 hybrid_clock.cc:648] HybridClock initialized: now 1781870544557533 us; error 46 us; skew 500 ppm
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:24.560875 14303 init.cc:377] Logged in from keytab as kudu/127.0.203.252@KRBTEST.COM (short username kudu)
I20260619 12:02:24.561434 14303 webserver.cc:460] Webserver started at http://127.0.203.252:42857/ using document root <none> and password file <none>
I20260619 12:02:24.561688 14303 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:24.561761 14303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:24.561915 14303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:24.563149 14303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data/instance:
uuid: "9733f6f89784412fb285253da7f1a679"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "e6245b705f4a10e9357a2c175cdccbdf"
server_key_iv: "d114b9dbf72994dd702afa36815d3902"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.563565 14303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/wal/instance:
uuid: "9733f6f89784412fb285253da7f1a679"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "e6245b705f4a10e9357a2c175cdccbdf"
server_key_iv: "d114b9dbf72994dd702afa36815d3902"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.565316 14303 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:02:24.565558 14296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } attempt: 2
I20260619 12:02:24.566107 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.565930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35528)
0619 12:02:24.566018 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.566066 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 12:02:24.566450 14318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:24.566752 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.566623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35542)
0619 12:02:24.566688 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.566722 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 12:02:24.566761 14303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
W20260619 12:02:24.566852 14296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:38329: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
I20260619 12:02:24.566911 14303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/wal
uuid: "9733f6f89784412fb285253da7f1a679"
format_stamp: "Formatted at 2026-06-19 12:02:24 on dist-test-slave-fbq8"
server_key: "e6245b705f4a10e9357a2c175cdccbdf"
server_key_iv: "d114b9dbf72994dd702afa36815d3902"
server_key_version: "encryptionkey@0"
I20260619 12:02:24.567021 14303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:24.593902 14221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } attempt: 2
I20260619 12:02:24.594457 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.594318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35558)
0619 12:02:24.594378 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.594425 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 12:02:24.594977 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.594872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35566)
0619 12:02:24.594925 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.594957 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 12:02:24.595168 14221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:38329: Network error: Client connection negotiation failed: client connection to 127.0.203.252:38329: connect: Connection refused (error 111)
I20260619 12:02:24.603329 14303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:24.605870 14303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:24.606051 14303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:24.611419 14303 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.252:38329
I20260619 12:02:24.611505 14370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.252:38329 every 8 connection(s)
I20260619 12:02:24.611935 14303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
I20260619 12:02:24.613322 14371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:24.615232 14371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:02:24.618742 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14303
I20260619 12:02:24.618834 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781870380032316-815-0/minicluster-data/master-2/wal/instance
I20260619 12:02:24.619041 815 external_mini_cluster.cc:1428] Setting key cc0e715a75603ac31f50063d76f6e1f5
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:02:24.627441 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.616212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.0.203.254:45167)
0619 12:02:24.616312 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.616315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.616524 (+ 209us) server_negotiation.cc:407] Connection header received
0619 12:02:24.617168 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.617169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.617183 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.617246 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.617967 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.618514 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.620278 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.620683 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.620764 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.623693 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.623695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.623695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.623704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.625232 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.625929 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.625930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.625931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.625995 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.626372 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.626374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.626374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.626567 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.626667 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.627178 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.627292 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:02:24.627556 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.616112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:45167 (local address 127.0.0.1:49464)
0619 12:02:24.616350 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.616398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.617055 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.617334 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.617342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.617645 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.617870 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.617874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.618703 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.618705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.620175 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.620178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.620809 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.620811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.621025 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.621698 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.621708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.623602 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.625382 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.625390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.625393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.625777 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.626104 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.626105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.626105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.626271 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.626769 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.626776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.626972 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.627122 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.627283 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":117,"threads_started":1}
I20260619 12:02:24.628572 14371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:02:24.630534 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.619570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.0.203.252:38329)
0619 12:02:24.619831 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.619837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.619861 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:02:24.619901 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.619903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.619927 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.620038 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.620898 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.621862 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.622789 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.623094 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.623215 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.626376 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.626387 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.626387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.626396 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.628772 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.629376 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.629381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.629381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.629464 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.629773 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.629775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.629776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.630011 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.630153 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.630327 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.630407 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":145,"threads_started":1}
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:24.637359 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.629101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:44789 (local address 127.0.0.1:43422)
0619 12:02:24.629282 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.629292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.629370 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.629492 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.629493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.629804 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.629865 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.629867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.630865 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.630866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.631493 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.631495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.632125 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.632126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.632181 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.632631 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.632666 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.633922 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.635792 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.635793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.635793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.636026 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.636246 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.636247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.636247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.636317 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.636950 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.636952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.637003 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.637110 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.637244 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 12:02:24.637356 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.629225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.203.253:44789)
0619 12:02:24.629298 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.629301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.629353 (+ 52us) server_negotiation.cc:407] Connection header received
0619 12:02:24.629427 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.629427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.629437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.629486 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.629991 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.630762 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.631632 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.632022 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.632116 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.634018 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.634021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.634021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.634030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.635674 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.636120 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.636121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.636121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.636173 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.636465 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.636467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.636468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.636622 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.636898 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.637163 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.637252 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 12:02:24.637908 14371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:02:24.640944 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.631591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.0.203.254:45167)
0619 12:02:24.631654 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.631657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.631664 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:02:24.631686 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.631687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.631696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.631749 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.632491 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.633193 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.634048 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.634278 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.634361 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.637211 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.637215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.637215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.637223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.639543 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.640066 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.640068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.640069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.640125 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.640343 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.640344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.640344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.640460 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.640712 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.640755 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.640833 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:02:24.649322 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.638252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.0.203.252:38329)
0619 12:02:24.638289 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.638291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.638493 (+ 202us) server_negotiation.cc:407] Connection header received
0619 12:02:24.638518 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.638519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.638536 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.638610 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.639228 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.639739 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.640582 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.641541 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.641658 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.645150 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.645158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.645158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.645165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.648052 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.648425 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.648426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.648426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.648489 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.648747 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.648748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.648749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.648869 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.648945 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.649088 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.649231 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":17}
I20260619 12:02:24.649322 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.638234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35576)
0619 12:02:24.638308 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.638322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.638384 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.638840 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.638842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.639021 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.639085 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.639087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.639864 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.639865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.640383 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.640384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.641656 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.641658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.641722 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.643437 (+ 1715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.643444 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.645014 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.648160 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.648161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.648162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.648337 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.648556 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.648557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.648558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.648617 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.648942 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.648943 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.648985 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.649065 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.649184 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:02:24.651463 14371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679: Bootstrap starting.
I20260619 12:02:24.652395 14371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:24.652770 14371 log.cc:824] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:24.653542 14371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679: No bootstrap required, opened a new log
I20260619 12:02:24.654079 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.643757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.0.203.253:44789)
0619 12:02:24.643805 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.643807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.643816 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:02:24.643843 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.643844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.643944 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.644026 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.644623 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.645524 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.647165 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.647756 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.647872 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.650836 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.650838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.650838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.650848 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.652544 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.653082 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.653083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.653084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.653160 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.653384 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.653385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.653385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.653532 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.653796 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.653841 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.653956 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:02:24.655853 14371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.656035 14371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:24.656103 14371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9733f6f89784412fb285253da7f1a679, State: Initialized, Role: FOLLOWER
I20260619 12:02:24.656246 14371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.656399 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:02:24.656627 14371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:24.656611 14377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.656839 14377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:02:24.657986 14296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } attempt: 3
W20260619 12:02:24.659406 14393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:02:24.659472 14393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:02:24.659847 14303 master_runner.cc:186] Error getting master registration for 127.0.203.254:45167: OK, instance_id { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" instance_seqno: 1781870544242021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:02:24.663272 14303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:02:24.666249 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.658285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35586)
0619 12:02:24.658325 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.658338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.658383 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.658581 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.658582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.658765 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.658824 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.658826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.659902 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.659903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.660418 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.660420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.660926 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.660928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.660981 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.661465 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.661472 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.663127 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.665077 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.665078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.665079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.665297 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.665518 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.665518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.665519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.665582 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.665898 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.665899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.665947 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.666016 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.666131 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 12:02:24.666887 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.658420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.203.252:38329)
0619 12:02:24.658462 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.658464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.658471 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:02:24.658492 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.658492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.658507 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.658566 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.658935 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.659741 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.660501 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.660799 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.660869 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.663228 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.663229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.663230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.663238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.664984 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.665391 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.665393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.665393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.665450 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.665645 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.665646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.665646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.665807 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.665870 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.666711 (+ 841us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.666796 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:02:24.669235 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: Bootstrap starting.
I20260619 12:02:24.670331 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.658824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.0.203.253:44789)
0619 12:02:24.658870 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.658872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.659701 (+ 829us) server_negotiation.cc:407] Connection header received
0619 12:02:24.659744 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.659745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.659757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.659808 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.661649 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.662400 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.664130 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.664430 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.664468 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.666683 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.666685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.666685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.666694 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.668705 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.669389 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.669391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.669391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.669458 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.669685 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.669686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.669687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.669831 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.669908 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.670141 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.670231 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 12:02:24.670506 14296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:24.670815 14225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.658761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.203.254:45167)
0619 12:02:24.658842 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.658844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.658854 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:02:24.658880 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.658881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.658893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.658951 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.659790 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.660711 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.664106 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.664403 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.664467 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.666684 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.666686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.666686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.666694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.668516 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.669375 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.669377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.669378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.669456 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.669745 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.669747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.669747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.670010 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.670116 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.670537 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.670719 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:02:24.670900 14296 log.cc:824] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:24.672262 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: No bootstrap required, opened a new log
I20260619 12:02:24.674971 14296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.675200 14296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:24.675298 14296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a3be83de1e49bdb13124f5fed0cfdb, State: Initialized, Role: FOLLOWER
I20260619 12:02:24.675478 14296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.675920 14296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:24.675854 14399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.676126 14399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [sys.catalog]: This master's current role is: FOLLOWER
W20260619 12:02:24.678336 14410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:02:24.678570 14410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:02:24.678452 14228 master_runner.cc:186] Error getting master registration for 127.0.203.254:45167: OK, instance_id { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" instance_seqno: 1781870544242021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:02:24.680585 14228 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:02:24.682185 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.659152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.0.203.252:38329)
0619 12:02:24.663593 (+ 4441us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.663595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.671171 (+ 7576us) server_negotiation.cc:407] Connection header received
0619 12:02:24.671273 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.671274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.671299 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.671342 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.672066 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.672956 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.674025 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.674475 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.674574 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.676324 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.676326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.676326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.676334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.680908 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.681310 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.681311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.681311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.681370 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.681576 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.681577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.681577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.681701 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.681805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.681958 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.682052 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4414,"thread_start_us":83,"threads_started":1}
I20260619 12:02:24.694407 14221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } attempt: 3
Jun 19 12:02:24 dist-test-slave-fbq8 krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:02:24.702572 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.694897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:38329 (local address 127.0.0.1:35606)
0619 12:02:24.695044 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:24.695060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:24.695101 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:24.695274 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:24.695276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:24.695459 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:24.695514 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.695515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.696206 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.696207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.696793 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:24.696795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.697179 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.697180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:24.697243 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.697738 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:24.697748 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:24.699029 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:24.701432 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.701433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.701433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.701613 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.701841 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:24.701842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:24.701842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:24.701908 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:24.702186 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:24.702188 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:24.702245 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 12:02:24.702348 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:02:24.702462 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 12:02:24.702898 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:24.695042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.0.203.252:38329)
0619 12:02:24.695143 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:24.695145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:24.695152 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:02:24.695174 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:24.695180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:24.695205 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:24.695271 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:24.695587 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.696128 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.696880 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:24.697089 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:24.697175 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:24.699110 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:24.699111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:24.699112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:24.699119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:24.701339 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.701705 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.701707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.701708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.701767 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:24.701970 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:24.701971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:24.701971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:24.702111 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:24.702185 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:24.702695 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 12:02:24.702817 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 12:02:24.704787 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: Bootstrap starting.
I20260619 12:02:24.705524 14221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:24.705861 14221 log.cc:824] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:24.706661 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: No bootstrap required, opened a new log
I20260619 12:02:24.708199 14221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.708336 14221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:24.708401 14221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ce1496c0f46509f8022423d95f91a, State: Initialized, Role: FOLLOWER
I20260619 12:02:24.708539 14221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.708873 14413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.708925 14221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:02:24.711617 14424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:02:24.711731 14424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:02:24.711062 14413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:02:24.714228 14153 master_runner.cc:263] No leader master found from master 9a3ce1496c0f46509f8022423d95f91a
I20260619 12:02:24.714274 14153 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 12:02:24.739326 14174 tablet.cc:2369] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:24.771512 14377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:24.771657 14377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.771947 14377 leader_election.cc:288] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3ce1496c0f46509f8022423d95f91a (127.0.203.254:45167), 42a3be83de1e49bdb13124f5fed0cfdb (127.0.203.253:44789)
I20260619 12:02:24.772508 14270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9733f6f89784412fb285253da7f1a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" is_pre_election: true
I20260619 12:02:24.772841 14270 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9733f6f89784412fb285253da7f1a679 in term 0.
I20260619 12:02:24.773157 14195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9733f6f89784412fb285253da7f1a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ce1496c0f46509f8022423d95f91a" is_pre_election: true
I20260619 12:02:24.773298 14195 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9733f6f89784412fb285253da7f1a679 in term 0.
I20260619 12:02:24.773581 14320 leader_election.cc:302] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a3be83de1e49bdb13124f5fed0cfdb, 9733f6f89784412fb285253da7f1a679; no voters:
I20260619 12:02:24.773770 14377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:02:24.773825 14377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:02:24.773849 14377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:24.775034 14377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.775159 14377 leader_election.cc:288] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [CANDIDATE]: Term 1 election: Requested vote from peers 9a3ce1496c0f46509f8022423d95f91a (127.0.203.254:45167), 42a3be83de1e49bdb13124f5fed0cfdb (127.0.203.253:44789)
I20260619 12:02:24.775516 14270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9733f6f89784412fb285253da7f1a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a3be83de1e49bdb13124f5fed0cfdb"
I20260619 12:02:24.775594 14270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:24.776661 14270 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9733f6f89784412fb285253da7f1a679 in term 1.
I20260619 12:02:24.777030 14195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9733f6f89784412fb285253da7f1a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ce1496c0f46509f8022423d95f91a"
I20260619 12:02:24.777112 14195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:24.778000 14195 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9733f6f89784412fb285253da7f1a679 in term 1.
I20260619 12:02:24.778228 14320 leader_election.cc:302] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a3be83de1e49bdb13124f5fed0cfdb, 9733f6f89784412fb285253da7f1a679; no voters:
I20260619 12:02:24.778452 14377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:24.778654 14377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [term 1 LEADER]: Becoming Leader. State: Replica: 9733f6f89784412fb285253da7f1a679, State: Running, Role: LEADER
I20260619 12:02:24.778745 14377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } }
I20260619 12:02:24.779142 14377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9733f6f89784412fb285253da7f1a679. Latest consensus state: current_term: 1 leader_uuid: "9733f6f89784412fb285253da7f1a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.779217 14377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:24.779556 14428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:24.779814 14428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:24.782152 14270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [term 1 FOLLOWER]: Refusing update from remote peer 9733f6f89784412fb285253da7f1a679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:24.782416 14195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [term 1 FOLLOWER]: Refusing update from remote peer 9733f6f89784412fb285253da7f1a679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:24.782789 14377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:24.784194 14413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9733f6f89784412fb285253da7f1a679. Latest consensus state: current_term: 1 leader_uuid: "9733f6f89784412fb285253da7f1a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.784289 14413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:02:24.784435 14377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:24.786762 14399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9733f6f89784412fb285253da7f1a679. Latest consensus state: current_term: 1 leader_uuid: "9733f6f89784412fb285253da7f1a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.786875 14399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:02:24.789100 14377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9733f6f89784412fb285253da7f1a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.789181 14377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:24.789600 14399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9733f6f89784412fb285253da7f1a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.789678 14399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:02:24.789935 14429 mvcc.cc:205] Tried to move back new op lower bound from 7298541751424761856 to 7298541751414329344. Current Snapshot: MvccSnapshot[applied={T|T < 7298541751424761856}]
I20260619 12:02:24.790764 14432 mvcc.cc:205] Tried to move back new op lower bound from 7298541751424761856 to 7298541751414329344. Current Snapshot: MvccSnapshot[applied={T|T < 7298541751424761856}]
I20260619 12:02:24.791179 14377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9733f6f89784412fb285253da7f1a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.791255 14377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:24.791740 14413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9733f6f89784412fb285253da7f1a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ce1496c0f46509f8022423d95f91a" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 45167 } } peers { permanent_uuid: "42a3be83de1e49bdb13124f5fed0cfdb" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 44789 } } peers { permanent_uuid: "9733f6f89784412fb285253da7f1a679" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 38329 } } }
I20260619 12:02:24.791822 14413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:02:24.792160 14430 mvcc.cc:205] Tried to move back new op lower bound from 7298541751424761856 to 7298541751414329344. Current Snapshot: MvccSnapshot[applied={T|T < 7298541751424761856}]
I20260619 12:02:24.793529 14428 catalog_manager.cc:1347] Generated new cluster ID: 76cfcddec6844a848a6f180836e973f3
I20260619 12:02:24.793578 14428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:02:24.808835 14428 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:02:24.809803 14428 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:02:24.819324 14428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9733f6f89784412fb285253da7f1a679: Generated new TSK 0
I20260619 12:02:24.819865 14428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:02:25.679086 14410 catalog_manager.cc:1259] Loaded cluster ID: 76cfcddec6844a848a6f180836e973f3
I20260619 12:02:25.679176 14410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: loading cluster ID for follower catalog manager: success
I20260619 12:02:25.680032 14410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: acquiring CA information for follower catalog manager: success
I20260619 12:02:25.680371 14410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42a3be83de1e49bdb13124f5fed0cfdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:02:25.712368 14424 catalog_manager.cc:1259] Loaded cluster ID: 76cfcddec6844a848a6f180836e973f3
I20260619 12:02:25.712505 14424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: loading cluster ID for follower catalog manager: success
I20260619 12:02:25.713568 14424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: acquiring CA information for follower catalog manager: success
I20260619 12:02:25.713925 14424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a3ce1496c0f46509f8022423d95f91a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:02:25.886032 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14153
I20260619 12:02:25.892395 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14228
I20260619 12:02:25.898885 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14303
2026-06-19T12:02:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3040 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:02:25 dist-test-slave-fbq8 krb5kdc[14444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:02:25 dist-test-slave-fbq8 krb5kdc[14444](info): set up 2 sockets
Jun 19 12:02:25 dist-test-slave-fbq8 krb5kdc[14444](info): commencing operation
krb5kdc: starting...
W20260619 12:02:26.969096 815 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:02:27 dist-test-slave-fbq8 krb5kdc[14444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:02:27.109956 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42767
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:42767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:27.205860 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:27.206108 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:27.206135 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:27.207388 14457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:27.207439 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:27.207453 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:27.207470 14457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:27.207482 14457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:27.209164 14457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:42767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:27.209448 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:27.209731 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:27.215485 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:27.215499 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:27.215600 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:27.215692 14457 server_base.cc:1034] running on GCE node
I20260619 12:02:27.216167 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:27.216566 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:27.217757 14457 hybrid_clock.cc:648] HybridClock initialized: now 1781870547217718 us; error 73 us; skew 500 ppm
Jun 19 12:02:27 dist-test-slave-fbq8 krb5kdc[14444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:27.221024 14457 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:27.221601 14457 webserver.cc:460] Webserver started at http://127.0.203.254:39171/ using document root <none> and password file <none>
I20260619 12:02:27.221868 14457 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:27.221917 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:27.222057 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:27.223238 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "089fae1d4ffe4ad5b9c8331ad4795128"
format_stamp: "Formatted at 2026-06-19 12:02:27 on dist-test-slave-fbq8"
server_key: "69b77fac49f14106956e283104b00c5e"
server_key_iv: "94ba36870a3049085142e587983a9012"
server_key_version: "encryptionkey@0"
I20260619 12:02:27.223623 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "089fae1d4ffe4ad5b9c8331ad4795128"
format_stamp: "Formatted at 2026-06-19 12:02:27 on dist-test-slave-fbq8"
server_key: "69b77fac49f14106956e283104b00c5e"
server_key_iv: "94ba36870a3049085142e587983a9012"
server_key_version: "encryptionkey@0"
I20260619 12:02:27.225284 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:02:27.226220 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:27.226681 14457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:27.226831 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "089fae1d4ffe4ad5b9c8331ad4795128"
format_stamp: "Formatted at 2026-06-19 12:02:27 on dist-test-slave-fbq8"
server_key: "69b77fac49f14106956e283104b00c5e"
server_key_iv: "94ba36870a3049085142e587983a9012"
server_key_version: "encryptionkey@0"
I20260619 12:02:27.226925 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:27.244206 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:27.247309 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:27.247499 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:27.252172 14457 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:42767
I20260619 12:02:27.252456 14524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:42767 every 8 connection(s)
I20260619 12:02:27.252652 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:27.253811 14525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:27.256635 14525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128: Bootstrap starting.
I20260619 12:02:27.256865 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14457
I20260619 12:02:27.257045 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:02:27.257318 815 external_mini_cluster.cc:1428] Setting key 439d558663db6b2cbf44021b2e9a2674
I20260619 12:02:27.257582 14525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:27.257951 14525 log.cc:824] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:27.258903 14525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128: No bootstrap required, opened a new log
I20260619 12:02:27.261188 14525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42767 } }
I20260619 12:02:27.261350 14525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:27.261409 14525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089fae1d4ffe4ad5b9c8331ad4795128, State: Initialized, Role: FOLLOWER
I20260619 12:02:27.261577 14525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42767 } }
I20260619 12:02:27.261675 14525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:02:27.261720 14525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:02:27.261771 14525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:27.262450 14525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42767 } }
I20260619 12:02:27.262588 14525 leader_election.cc:302] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089fae1d4ffe4ad5b9c8331ad4795128; no voters:
I20260619 12:02:27.262832 14525 leader_election.cc:288] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:02:27.262926 14530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:27.263142 14525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:27.263181 14530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [term 1 LEADER]: Becoming Leader. State: Replica: 089fae1d4ffe4ad5b9c8331ad4795128, State: Running, Role: LEADER
I20260619 12:02:27.263484 14530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42767 } }
I20260619 12:02:27.264011 14531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42767 } } }
I20260619 12:02:27.264109 14531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:27.264405 14539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:27.264540 14532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089fae1d4ffe4ad5b9c8331ad4795128. Latest consensus state: current_term: 1 leader_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089fae1d4ffe4ad5b9c8331ad4795128" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 42767 } } }
I20260619 12:02:27.264632 14532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128 [sys.catalog]: This master's current role is: LEADER
Jun 19 12:02:27 dist-test-slave-fbq8 krb5kdc[14444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:27.265166 14539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:27.267283 14539 catalog_manager.cc:1347] Generated new cluster ID: 9e751acd201d4d2b8d37ddf0688d049c
I20260619 12:02:27.267350 14539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:02:27.270008 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:27.257996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.203.254:42767)
0619 12:02:27.258289 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:27.258301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:27.258346 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:02:27.259073 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:27.259083 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:27.259153 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:27.259311 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:27.259985 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:27.261029 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:27.261968 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:27.262415 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:27.262534 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:27.265730 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:27.265734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:27.265735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:27.265750 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:27.268052 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:27.268515 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:27.268519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:27.268520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:27.268680 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:27.268981 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:27.268983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:27.268984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:27.269325 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:27.269513 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:27.269643 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 12:02:27.269795 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":120,"threads_started":1}
I20260619 12:02:27.271273 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:02:27.274873 14539 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:02:27.275442 14539 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:02:27.278084 14539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089fae1d4ffe4ad5b9c8331ad4795128: Generated new TSK 0
I20260619 12:02:27.278280 14539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:02:27.326354 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14457
2026-06-19T12:02:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1424 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:02:27 dist-test-slave-fbq8 krb5kdc[14552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:02:27 dist-test-slave-fbq8 krb5kdc[14552](info): set up 2 sockets
Jun 19 12:02:27 dist-test-slave-fbq8 krb5kdc[14552](info): commencing operation
krb5kdc: starting...
W20260619 12:02:28.400933 815 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:02:28 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:02:28.533599 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:32777
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:28.630504 14565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:28.630750 14565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:28.630774 14565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:28.631990 14565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:28.632037 14565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:28.632051 14565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:28.632064 14565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:28.632076 14565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:28.633692 14565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:28.633961 14565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:28.634238 14565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:28.640609 14570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:28.640750 14571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:28.640623 14573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:28.640940 14565 server_base.cc:1034] running on GCE node
I20260619 12:02:28.641355 14565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:28.641705 14565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:28.642916 14565 hybrid_clock.cc:648] HybridClock initialized: now 1781870548642888 us; error 46 us; skew 500 ppm
Jun 19 12:02:28 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:28.646188 14565 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:28.646778 14565 webserver.cc:460] Webserver started at http://127.0.203.254:46141/ using document root <none> and password file <none>
I20260619 12:02:28.647037 14565 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:28.647106 14565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:28.647255 14565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:28.648485 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "9dabfc72a7ac49d182e07b709e4b9af1"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "6fadb05c19f5ed1178aeb91c89fb8e8d"
server_key_iv: "7592e2bebcbbaf5354b005fa4ac1ecdb"
server_key_version: "encryptionkey@0"
I20260619 12:02:28.648955 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "9dabfc72a7ac49d182e07b709e4b9af1"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "6fadb05c19f5ed1178aeb91c89fb8e8d"
server_key_iv: "7592e2bebcbbaf5354b005fa4ac1ecdb"
server_key_version: "encryptionkey@0"
I20260619 12:02:28.650681 14565 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:28.651782 14580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:28.652127 14565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:28.652273 14565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "9dabfc72a7ac49d182e07b709e4b9af1"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "6fadb05c19f5ed1178aeb91c89fb8e8d"
server_key_iv: "7592e2bebcbbaf5354b005fa4ac1ecdb"
server_key_version: "encryptionkey@0"
I20260619 12:02:28.652357 14565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:28.686669 14565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:28.689169 14565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:28.689342 14565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:28.694581 14565 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:32777
I20260619 12:02:28.694693 14632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:32777 every 8 connection(s)
I20260619 12:02:28.695060 14565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:28.696231 14633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:28.698793 14633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Bootstrap starting.
I20260619 12:02:28.699378 14633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:28.699632 14633 log.cc:824] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:28.700479 14633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: No bootstrap required, opened a new log
I20260619 12:02:28.700796 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14565
I20260619 12:02:28.700912 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:02:28.701134 815 external_mini_cluster.cc:1428] Setting key 45879a7633dfc73b52849336a3d1a4a7
I20260619 12:02:28.702908 14633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:28.703074 14633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:28.703099 14633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dabfc72a7ac49d182e07b709e4b9af1, State: Initialized, Role: FOLLOWER
I20260619 12:02:28.703258 14633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:28.703356 14633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:02:28.703397 14633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:02:28.703449 14633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:28.704408 14633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:28.704541 14633 leader_election.cc:302] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dabfc72a7ac49d182e07b709e4b9af1; no voters:
I20260619 12:02:28.704807 14633 leader_election.cc:288] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:02:28.704833 14638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:28.704957 14638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 LEADER]: Becoming Leader. State: Replica: 9dabfc72a7ac49d182e07b709e4b9af1, State: Running, Role: LEADER
I20260619 12:02:28.705137 14633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:28.705147 14638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:28.705710 14640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dabfc72a7ac49d182e07b709e4b9af1. Latest consensus state: current_term: 1 leader_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } } }
I20260619 12:02:28.705873 14640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:28.706676 14639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } } }
I20260619 12:02:28.706835 14639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:28.707100 14647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:28.707753 14647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:28.709563 14647 catalog_manager.cc:1347] Generated new cluster ID: bb95aa71a89a42869fd105c06f68c2b5
I20260619 12:02:28.709626 14647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:02:28 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:28.715586 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:28.701784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.0.203.254:32777)
0619 12:02:28.702000 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:28.702010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:28.702042 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:02:28.702905 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:28.702916 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:28.702987 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:28.703194 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:28.704018 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:28.705293 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:28.706377 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:28.707142 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:28.707314 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:28.711036 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:28.711042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:28.711042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:28.711058 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:28.713183 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:28.713591 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:28.713594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:28.713594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:28.714097 (+ 503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:28.714286 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:28.714287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:28.714287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:28.714602 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:28.714749 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:28.715186 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 12:02:28.715336 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260619 12:02:28.720948 14647 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:02:28.721683 14647 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:02:28.729475 14647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Generated new TSK 0
I20260619 12:02:28.729682 14647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:02:28.764019 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:32777
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:28.857259 14659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:28.857509 14659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:28.857548 14659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:28.858736 14659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:28.858800 14659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:28.858847 14659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:02:28.860379 14659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:28.860707 14659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:28.860946 14659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:28.866245 14665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:28.866279 14659 server_base.cc:1034] running on GCE node
W20260619 12:02:28.866285 14664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:28.866245 14667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:28.866739 14659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:28.867008 14659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:28.868211 14659 hybrid_clock.cc:648] HybridClock initialized: now 1781870548868214 us; error 49 us; skew 500 ppm
Jun 19 12:02:28 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:28.870976 14659 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:02:28.871443 14659 webserver.cc:460] Webserver started at http://127.0.203.193:32813/ using document root <none> and password file <none>
I20260619 12:02:28.871696 14659 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:28.871745 14659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:28.871886 14659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:28.873055 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "01080459e55543dcad28d57fd5048cf6"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "03b84f95831b76ac186643ab9704756d"
server_key_iv: "15d58774682b1c3fb4dc031556b81d2c"
server_key_version: "encryptionkey@0"
I20260619 12:02:28.873405 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "01080459e55543dcad28d57fd5048cf6"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "03b84f95831b76ac186643ab9704756d"
server_key_iv: "15d58774682b1c3fb4dc031556b81d2c"
server_key_version: "encryptionkey@0"
I20260619 12:02:28.874859 14659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:02:28.875761 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:28.875991 14659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:28.876098 14659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "01080459e55543dcad28d57fd5048cf6"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "03b84f95831b76ac186643ab9704756d"
server_key_iv: "15d58774682b1c3fb4dc031556b81d2c"
server_key_version: "encryptionkey@0"
I20260619 12:02:28.876186 14659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:28.887984 14659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:28.890583 14659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:28.890740 14659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:28.891010 14659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:28.891356 14659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:28.891412 14659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:28.891456 14659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:28.891484 14659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:28.898037 14659 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:41603
I20260619 12:02:28.898149 14787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:41603 every 8 connection(s)
I20260619 12:02:28.898469 14659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:02:28.900408 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14659
I20260619 12:02:28.900501 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:02:28.900751 815 external_mini_cluster.cc:1428] Setting key 299265bfa9315c86324c6981bd2e5f47
Jun 19 12:02:28 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 12:02:28.910926 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:28.899285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:36271 (local address 127.0.203.254:32777)
0619 12:02:28.899345 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:28.899347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:28.899565 (+ 218us) server_negotiation.cc:407] Connection header received
0619 12:02:28.900091 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:28.900092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:28.900105 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:28.900148 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:28.901442 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:28.902196 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:28.903681 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:28.903917 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:28.903994 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:28.907215 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:28.907218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:28.907218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:28.907231 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:28.909158 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:28.909704 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:28.909705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:28.909706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:28.909780 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:28.910144 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:28.910146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:28.910146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:28.910338 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:28.910424 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:28.910719 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 12:02:28.910824 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 12:02:28.911235 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:28.899232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:36271)
0619 12:02:28.899407 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:28.899452 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:28.899975 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:28.900211 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:28.900214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:28.901051 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:28.901283 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:28.901289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:28.902318 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:28.902319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:28.903571 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:28.903575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:28.904018 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:28.904019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:28.904179 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:28.904910 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:28.904922 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:28.907099 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:28.909285 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:28.909297 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:28.909301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:28.909607 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:28.909857 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:28.909858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:28.909859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:28.909993 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:28.910413 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:28.910421 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:28.910580 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 12:02:28.910697 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 12:02:28.910889 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260619 12:02:28.911798 14788 heartbeater.cc:347] Connected to a master server at 127.0.203.254:32777
I20260619 12:02:28.911962 14788 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:28.912191 14788 heartbeater.cc:511] Master 127.0.203.254:32777 requested a full tablet report, sending...
I20260619 12:02:28.912920 14597 ts_manager.cc:194] Registered new tserver with Master: 01080459e55543dcad28d57fd5048cf6 (127.0.203.193:41603)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
I20260619 12:02:28.913708 14597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:36271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:02:28.934299 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:32777
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:29.025244 14793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:29.025478 14793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:29.025506 14793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:29.026698 14793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:29.026749 14793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:29.026779 14793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:02:29.028285 14793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:29.028506 14793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:29.028762 14793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:29.033918 14801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.033970 14798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.033918 14799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:29.034150 14793 server_base.cc:1034] running on GCE node
I20260619 12:02:29.034391 14793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:29.034639 14793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:29.035832 14793 hybrid_clock.cc:648] HybridClock initialized: now 1781870549035811 us; error 32 us; skew 500 ppm
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:29.038682 14793 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:02:29.039198 14793 webserver.cc:460] Webserver started at http://127.0.203.194:35203/ using document root <none> and password file <none>
I20260619 12:02:29.039443 14793 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:29.039508 14793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:29.039647 14793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:29.040835 14793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "e28590f0a1334f5087209bb738de2bf4"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "fdfbe191f790e68d8d4b08278f3364ef"
server_key_iv: "4c6dbed50301ff1a3dddd007ddd8348c"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.041191 14793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "e28590f0a1334f5087209bb738de2bf4"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "fdfbe191f790e68d8d4b08278f3364ef"
server_key_iv: "4c6dbed50301ff1a3dddd007ddd8348c"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.042554 14793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:02:29.043385 14808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.043625 14793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:29.043763 14793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "e28590f0a1334f5087209bb738de2bf4"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "fdfbe191f790e68d8d4b08278f3364ef"
server_key_iv: "4c6dbed50301ff1a3dddd007ddd8348c"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.043844 14793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:29.067844 14793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:29.070384 14793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:29.070569 14793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:29.070907 14793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:29.071295 14793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:29.071352 14793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.071396 14793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:29.071424 14793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.077471 14793 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:33787
I20260619 12:02:29.077513 14921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:33787 every 8 connection(s)
I20260619 12:02:29.077839 14793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:02:29.080013 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14793
I20260619 12:02:29.080150 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:02:29.080354 815 external_mini_cluster.cc:1428] Setting key d7d1cbbbddbacca7a761220da5194ec5
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:29.088846 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.078566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:46037 (local address 127.0.203.254:32777)
0619 12:02:29.078614 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.078616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.078800 (+ 184us) server_negotiation.cc:407] Connection header received
0619 12:02:29.079320 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.079321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.079330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.079396 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:29.080471 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.081141 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.082211 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.082517 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.082585 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.085445 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:29.085446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:29.085447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:29.085454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:29.087068 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.087630 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.087631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.087632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.087693 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.088007 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.088007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.088008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.088175 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:29.088230 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.088632 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.088774 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 12:02:29.089128 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.078540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:46037)
0619 12:02:29.078689 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.078730 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.079237 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.079450 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.079453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.080105 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:29.080373 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.080378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.081253 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.081254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.082097 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.082099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.082620 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.082621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.082778 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.083373 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:29.083384 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:29.085328 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:29.087163 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.087172 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.087175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.087532 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.087771 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.087772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.087772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.087930 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.088250 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:29.088257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:29.088444 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.088596 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.088829 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
I20260619 12:02:29.089831 14922 heartbeater.cc:347] Connected to a master server at 127.0.203.254:32777
I20260619 12:02:29.089941 14922 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:29.090152 14922 heartbeater.cc:511] Master 127.0.203.254:32777 requested a full tablet report, sending...
I20260619 12:02:29.090595 14597 ts_manager.cc:194] Registered new tserver with Master: e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787)
I20260619 12:02:29.091009 14597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:46037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:02:29.108927 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:32777
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:29.194497 14927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:29.194708 14927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:29.194732 14927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:29.195812 14927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:29.195848 14927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:29.195874 14927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:02:29.197278 14927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:29.197463 14927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:29.197652 14927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:29.202767 14933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.202801 14935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.202771 14932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:29.202950 14927 server_base.cc:1034] running on GCE node
I20260619 12:02:29.203339 14927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:29.203612 14927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:29.204767 14927 hybrid_clock.cc:648] HybridClock initialized: now 1781870549204756 us; error 38 us; skew 500 ppm
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:29.207680 14927 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:02:29.208077 14927 webserver.cc:460] Webserver started at http://127.0.203.195:38973/ using document root <none> and password file <none>
I20260619 12:02:29.208314 14927 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:29.208359 14927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:29.208477 14927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:29.209611 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "20f3b0b5e57344d78ef2d16ee15ae343"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "f1125ecb17560a0435ade5c690b029e0"
server_key_iv: "e9a966370195af1d7d7b7e37bc629741"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.209964 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "20f3b0b5e57344d78ef2d16ee15ae343"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "f1125ecb17560a0435ade5c690b029e0"
server_key_iv: "e9a966370195af1d7d7b7e37bc629741"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.211350 14927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:02:29.212181 14942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.212388 14927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:29.212493 14927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "20f3b0b5e57344d78ef2d16ee15ae343"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "f1125ecb17560a0435ade5c690b029e0"
server_key_iv: "e9a966370195af1d7d7b7e37bc629741"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.212611 14927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:29.227277 14927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:29.229743 14927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:29.229915 14927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:29.230176 14927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:29.230499 14927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:29.230551 14927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.230594 14927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:29.230620 14927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.236682 14927 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:37117
I20260619 12:02:29.236773 15055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:37117 every 8 connection(s)
I20260619 12:02:29.237118 14927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:29.244571 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 14927
I20260619 12:02:29.244683 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:02:29.244874 815 external_mini_cluster.cc:1428] Setting key db3874e13d7c202e1f87cfecba9a03ca
I20260619 12:02:29.246924 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.237908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:56859 (local address 127.0.203.254:32777)
0619 12:02:29.237973 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.237976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.238228 (+ 252us) server_negotiation.cc:407] Connection header received
0619 12:02:29.238755 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.238756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.238767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.238859 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:29.239781 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.240287 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.241378 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.241579 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.241643 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.243900 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:29.243901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:29.243901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:29.243908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:29.245685 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.246138 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.246139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.246139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.246193 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.246373 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.246373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.246373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.246466 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:29.246510 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.246742 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.246839 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 12:02:29.247291 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.237893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:56859)
0619 12:02:29.238073 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.238112 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.238659 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.238876 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.238879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.239477 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:29.239675 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.239680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.240373 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.240374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.241288 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.241291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.241675 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.241677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.241792 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.242292 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:29.242300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:29.243812 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:29.245762 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.245769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.245772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.246024 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.246240 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.246241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.246241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.246337 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.246515 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:29.246519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:29.246646 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.246810 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.246990 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260619 12:02:29.247800 15056 heartbeater.cc:347] Connected to a master server at 127.0.203.254:32777
I20260619 12:02:29.247925 15056 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:29.248142 15056 heartbeater.cc:511] Master 127.0.203.254:32777 requested a full tablet report, sending...
I20260619 12:02:29.248541 14597 ts_manager.cc:194] Registered new tserver with Master: 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
I20260619 12:02:29.248945 14597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:56859
I20260619 12:02:29.257040 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:02:29.265013 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.258590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.0.203.254:32777)
0619 12:02:29.258655 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.258658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.258800 (+ 142us) server_negotiation.cc:407] Connection header received
0619 12:02:29.258858 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.258858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.258868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.258916 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:29.259595 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.260079 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.260896 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.261075 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.261138 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.262375 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:29.262377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:29.262377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:29.262383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:29.263799 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.264250 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.264250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.264251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.264303 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.264452 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.264452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.264452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.264601 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:29.264747 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.264862 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.264925 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 12:02:29.266918 14597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:02:29.268567 14597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:02:29.275633 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.271560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.0.203.193:41603)
0619 12:02:29.272102 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.272119 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.272129 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:02:29.272170 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.272173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.272190 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.272269 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:29.272480 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.273561 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.274695 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.275203 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.275290 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.275354 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.275499 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.275538 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":75,"threads_started":1}
I20260619 12:02:29.275625 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.271300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41603 (local address 127.0.0.1:44618)
0619 12:02:29.271668 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.271675 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.271708 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.272351 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.272352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.272354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:29.272392 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.272394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.273657 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.273658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.274609 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.274612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.275293 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.275293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.275370 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.275384 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.275481 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.275537 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
I20260619 12:02:29.275902 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.271300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:37117 (local address 127.0.0.1:32792)
0619 12:02:29.271531 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.271541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.271597 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.271947 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.271949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.271952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:29.272095 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.272099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.273476 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.273477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.274656 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.274659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.275641 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.275641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.275708 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.275712 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.275763 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.275817 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":38,"thread_start_us":107,"threads_started":1}
I20260619 12:02:29.276114 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.271259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:33787 (local address 127.0.0.1:49520)
0619 12:02:29.271445 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.271484 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.271558 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.271974 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.271976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.271979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:29.272098 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.272101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.273279 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.273280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.274658 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.274659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.275639 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.275640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.275863 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.275866 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.275911 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.276027 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
I20260619 12:02:29.276134 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.271561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.0.203.195:37117)
0619 12:02:29.271774 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.271779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.271789 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:02:29.271830 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.271832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.271844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.271962 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:29.272182 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.273363 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.274766 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.275479 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.275557 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.275603 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.276017 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.276057 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":133,"threads_started":1}
I20260619 12:02:29.276330 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.271353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.0.203.194:33787)
0619 12:02:29.271795 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.271799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.271810 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:02:29.271847 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.271848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.271861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.271961 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:29.272192 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.273161 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.274765 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.275479 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.275548 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.275599 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.276224 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.276266 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":90,"threads_started":1}
I20260619 12:02:29.276445 14722 tablet_service.cc:1467] Processing CreateTablet for tablet 6d29bbacfab64105ac68e7275e295427 (DEFAULT_TABLE table=test-table [id=85060eb8828c4c06a20db6cdb2d6cb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:29.276868 14722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d29bbacfab64105ac68e7275e295427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:29.277345 14856 tablet_service.cc:1467] Processing CreateTablet for tablet 6d29bbacfab64105ac68e7275e295427 (DEFAULT_TABLE table=test-table [id=85060eb8828c4c06a20db6cdb2d6cb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:29.277452 14990 tablet_service.cc:1467] Processing CreateTablet for tablet 6d29bbacfab64105ac68e7275e295427 (DEFAULT_TABLE table=test-table [id=85060eb8828c4c06a20db6cdb2d6cb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:29.277643 14856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d29bbacfab64105ac68e7275e295427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:29.277773 14990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d29bbacfab64105ac68e7275e295427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:29.280702 15074 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Bootstrap starting.
I20260619 12:02:29.280791 15073 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Bootstrap starting.
I20260619 12:02:29.281080 15075 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Bootstrap starting.
I20260619 12:02:29.281426 15073 tablet_bootstrap.cc:654] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:29.281440 15074 tablet_bootstrap.cc:654] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:29.281747 15075 tablet_bootstrap.cc:654] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:29.282014 15073 log.cc:824] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:29.282054 15074 log.cc:824] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:29.282336 15075 log.cc:824] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:29.282743 15073 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: No bootstrap required, opened a new log
I20260619 12:02:29.282799 15073 ts_tablet_manager.cc:1397] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:29.283010 15075 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: No bootstrap required, opened a new log
I20260619 12:02:29.283089 15075 ts_tablet_manager.cc:1397] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:29.284416 15073 raft_consensus.cc:348] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.284567 15073 raft_consensus.cc:374] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:29.284591 15073 raft_consensus.cc:729] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28590f0a1334f5087209bb738de2bf4, State: Initialized, Role: FOLLOWER
I20260619 12:02:29.284534 15075 raft_consensus.cc:348] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.284765 15075 raft_consensus.cc:374] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:29.284735 15073 consensus_queue.cc:260] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.284819 15075 raft_consensus.cc:729] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f3b0b5e57344d78ef2d16ee15ae343, State: Initialized, Role: FOLLOWER
I20260619 12:02:29.284955 15075 consensus_queue.cc:260] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.285017 15073 ts_tablet_manager.cc:1428] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:29.285104 15074 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: No bootstrap required, opened a new log
I20260619 12:02:29.285180 15074 ts_tablet_manager.cc:1397] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:02:29.285218 14922 heartbeater.cc:503] Master 127.0.203.254:32777 was elected leader, sending a full tablet report...
I20260619 12:02:29.285233 15075 ts_tablet_manager.cc:1428] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:29.285351 15056 heartbeater.cc:503] Master 127.0.203.254:32777 was elected leader, sending a full tablet report...
I20260619 12:02:29.287012 15074 raft_consensus.cc:348] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.287184 15074 raft_consensus.cc:374] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:29.287231 15074 raft_consensus.cc:729] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01080459e55543dcad28d57fd5048cf6, State: Initialized, Role: FOLLOWER
I20260619 12:02:29.287341 15074 consensus_queue.cc:260] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.287662 15074 ts_tablet_manager.cc:1428] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 12:02:29.287714 14788 heartbeater.cc:503] Master 127.0.203.254:32777 was elected leader, sending a full tablet report...
W20260619 12:02:29.328428 14923 tablet.cc:2369] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:29.357795 15081 raft_consensus.cc:482] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:29.357949 15081 raft_consensus.cc:504] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.358435 15081 leader_election.cc:288] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
I20260619 12:02:29.361670 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.358865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:59635 (local address 127.0.203.195:37117)
0619 12:02:29.358936 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.358939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.358948 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:02:29.358975 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.358976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.358984 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.359075 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:29.359253 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.359793 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.360855 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.361268 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.361368 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.361377 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.361542 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.361583 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 12:02:29.361670 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.358788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:56449 (local address 127.0.203.194:33787)
0619 12:02:29.358842 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.358844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.359140 (+ 296us) server_negotiation.cc:407] Connection header received
0619 12:02:29.359212 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.359213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.359223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.359286 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:29.359432 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.359892 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.360931 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.361320 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.361403 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.361412 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.361561 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.361600 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:02:29.361676 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.358702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:37117 (local address 127.0.203.193:59635)
0619 12:02:29.358747 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.358759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.358831 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.359081 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.359083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.359085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:29.359174 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.359176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.359884 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.359885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.360752 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.360755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.361367 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.361368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.361461 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.361464 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.361535 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.361582 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 12:02:29.361685 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.358702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:33787 (local address 127.0.203.193:56449)
0619 12:02:29.359067 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.359080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.359146 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.359290 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.359291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.359292 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:29.359366 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.359367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.359986 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.359987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.360845 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.360847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.361402 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.361403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.361476 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.361480 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.361544 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.361590 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":179,"thread_start_us":123,"threads_started":1}
I20260619 12:02:29.362144 15010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d29bbacfab64105ac68e7275e295427" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" is_pre_election: true
I20260619 12:02:29.362143 14876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d29bbacfab64105ac68e7275e295427" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28590f0a1334f5087209bb738de2bf4" is_pre_election: true
I20260619 12:02:29.362317 15010 raft_consensus.cc:2454] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 0.
I20260619 12:02:29.362317 14876 raft_consensus.cc:2454] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 0.
I20260619 12:02:29.362586 14676 leader_election.cc:302] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6, 20f3b0b5e57344d78ef2d16ee15ae343; no voters:
I20260619 12:02:29.362802 15081 raft_consensus.cc:2781] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:02:29.362895 15081 raft_consensus.cc:482] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:02:29.362931 15081 raft_consensus.cc:3037] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:29.363745 15081 raft_consensus.cc:504] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.363921 15081 leader_election.cc:288] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 election: Requested vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
I20260619 12:02:29.364138 14876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d29bbacfab64105ac68e7275e295427" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28590f0a1334f5087209bb738de2bf4"
I20260619 12:02:29.364146 15010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d29bbacfab64105ac68e7275e295427" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f3b0b5e57344d78ef2d16ee15ae343"
I20260619 12:02:29.364236 14876 raft_consensus.cc:3037] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:29.364238 15010 raft_consensus.cc:3037] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:29.365101 14876 raft_consensus.cc:2454] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 1.
I20260619 12:02:29.365114 15010 raft_consensus.cc:2454] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 1.
I20260619 12:02:29.365347 14678 leader_election.cc:302] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6, e28590f0a1334f5087209bb738de2bf4; no voters:
I20260619 12:02:29.365485 15081 raft_consensus.cc:2781] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:29.365705 15081 raft_consensus.cc:686] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 1 LEADER]: Becoming Leader. State: Replica: 01080459e55543dcad28d57fd5048cf6, State: Running, Role: LEADER
I20260619 12:02:29.365821 15081 consensus_queue.cc:237] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.366770 14597 catalog_manager.cc:5495] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01080459e55543dcad28d57fd5048cf6 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "01080459e55543dcad28d57fd5048cf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } health_report { overall_health: HEALTHY } } }
I20260619 12:02:29.394232 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.391567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.0.203.193:41603)
0619 12:02:29.391654 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.391656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.391673 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:02:29.391711 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.391712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.391731 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.391803 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:02:29.391980 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.392537 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.393369 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.393567 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.393634 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.393818 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:02:29.393925 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:02:29.393989 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.394073 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.394092 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:02:29.396385 15010 raft_consensus.cc:1261] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Refusing update from remote peer 01080459e55543dcad28d57fd5048cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:29.396423 14876 raft_consensus.cc:1261] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Refusing update from remote peer 01080459e55543dcad28d57fd5048cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:29.396894 15083 consensus_queue.cc:1035] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:29.397058 15081 consensus_queue.cc:1035] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 12:02:29.399362 14789 tablet.cc:2369] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:29.399379 15086 mvcc.cc:205] Tried to move back new op lower bound from 7298541770324148224 to 7298541770203295744. Current Snapshot: MvccSnapshot[applied={T|T < 7298541770324148224}]
I20260619 12:02:29.399750 15087 mvcc.cc:205] Tried to move back new op lower bound from 7298541770324148224 to 7298541770203295744. Current Snapshot: MvccSnapshot[applied={T|T < 7298541770324148224}]
I20260619 12:02:29.402311 14597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:29.402408 14597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:29.403172 14597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Sending DeleteTablet for 3 replicas of tablet 6d29bbacfab64105ac68e7275e295427
I20260619 12:02:29.403512 14990 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d29bbacfab64105ac68e7275e295427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:29 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:32792
I20260619 12:02:29.403584 14722 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d29bbacfab64105ac68e7275e295427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:29 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44618
I20260619 12:02:29.403553 14856 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d29bbacfab64105ac68e7275e295427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:29 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:49520
I20260619 12:02:29.403820 15098 tablet_replica.cc:330] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: stopping tablet replica
I20260619 12:02:29.403882 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14659
I20260619 12:02:29.403884 15096 tablet_replica.cc:330] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: stopping tablet replica
I20260619 12:02:29.403930 15098 raft_consensus.cc:2229] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:29.404462 15098 raft_consensus.cc:2258] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:29.404685 15096 raft_consensus.cc:2229] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:29.404800 15096 raft_consensus.cc:2258] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:29.405058 15098 ts_tablet_manager.cc:1905] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:29.405421 15096 ts_tablet_manager.cc:1905] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:29.407033 15096 ts_tablet_manager.cc:1918] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:29.407191 15096 log.cc:1196] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/6d29bbacfab64105ac68e7275e295427
I20260619 12:02:29.407195 15098 ts_tablet_manager.cc:1918] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:29.407265 15098 log.cc:1196] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/6d29bbacfab64105ac68e7275e295427
I20260619 12:02:29.407500 15096 ts_tablet_manager.cc:1939] T 6d29bbacfab64105ac68e7275e295427 P 20f3b0b5e57344d78ef2d16ee15ae343: Deleting consensus metadata
I20260619 12:02:29.407560 15098 ts_tablet_manager.cc:1939] T 6d29bbacfab64105ac68e7275e295427 P e28590f0a1334f5087209bb738de2bf4: Deleting consensus metadata
I20260619 12:02:29.407955 14584 catalog_manager.cc:4841] TS e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787): tablet 6d29bbacfab64105ac68e7275e295427 (table test-table [id=85060eb8828c4c06a20db6cdb2d6cb2e]) successfully deleted
I20260619 12:02:29.407958 14582 catalog_manager.cc:4841] TS 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117): tablet 6d29bbacfab64105ac68e7275e295427 (table test-table [id=85060eb8828c4c06a20db6cdb2d6cb2e]) successfully deleted
W20260619 12:02:29.410604 14583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:41603 (error 108)
I20260619 12:02:29.410846 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14793
I20260619 12:02:29.411923 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.411777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41603 (local address 127.0.0.1:44634)
0619 12:02:29.411841 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.411884 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 12:02:29.412075 14583 catalog_manager.cc:4584] TS 01080459e55543dcad28d57fd5048cf6 (127.0.203.193:41603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d29bbacfab64105ac68e7275e295427: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41603: connect: Connection refused (error 111)
I20260619 12:02:29.415576 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14927
I20260619 12:02:29.420825 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 14565
I20260619 12:02:29.426815 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:32777
--webserver_interface=127.0.203.254
--webserver_port=46141
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:29.508240 15100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:29.508498 15100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:29.508535 15100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:29.509761 15100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:29.509838 15100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:29.509863 15100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:29.509886 15100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:29.509912 15100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:29.511420 15100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:29.511715 15100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:29.511945 15100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:29.521135 15107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.521134 15106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.521288 15109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:29.522634 15100 server_base.cc:1034] running on GCE node
I20260619 12:02:29.522964 15100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:29.523406 15100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:29.524601 15100 hybrid_clock.cc:648] HybridClock initialized: now 1781870549524590 us; error 32 us; skew 500 ppm
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:29.527698 15100 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:29.528196 15100 webserver.cc:460] Webserver started at http://127.0.203.254:46141/ using document root <none> and password file <none>
I20260619 12:02:29.528529 15100 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:29.528674 15100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:29.530207 15100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:29.530951 15116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.531208 15100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:29.531349 15100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "9dabfc72a7ac49d182e07b709e4b9af1"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "6fadb05c19f5ed1178aeb91c89fb8e8d"
server_key_iv: "7592e2bebcbbaf5354b005fa4ac1ecdb"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.531903 15100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:29.549726 15100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:29.552434 15100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:29.552613 15100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:29.557241 15100 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:32777
I20260619 12:02:29.557292 15168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:32777 every 8 connection(s)
I20260619 12:02:29.557720 15100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:29.559439 15169 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:02:29.560194 15169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Bootstrap starting.
I20260619 12:02:29.562193 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15100
I20260619 12:02:29.562474 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:41603
--local_ip_for_outbound_sockets=127.0.203.193
--tserver_master_addrs=127.0.203.254:32777
--webserver_port=32813
--webserver_interface=127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:02:29.570554 15169 log.cc:824] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:29.574204 15169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:02:29.574661 15169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Bootstrap complete.
I20260619 12:02:29.578755 15169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:29.579255 15169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dabfc72a7ac49d182e07b709e4b9af1, State: Initialized, Role: FOLLOWER
I20260619 12:02:29.579492 15169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:29.579583 15169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:02:29.579632 15169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:02:29.579674 15169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:02:29.581224 15169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:29.581342 15169 leader_election.cc:302] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dabfc72a7ac49d182e07b709e4b9af1; no voters:
I20260619 12:02:29.581646 15169 leader_election.cc:288] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:02:29.581745 15173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:02:29.581934 15173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 LEADER]: Becoming Leader. State: Replica: 9dabfc72a7ac49d182e07b709e4b9af1, State: Running, Role: LEADER
I20260619 12:02:29.581945 15169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:29.583396 15173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:29.583978 15175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } } }
I20260619 12:02:29.584120 15174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dabfc72a7ac49d182e07b709e4b9af1. Latest consensus state: current_term: 2 leader_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } } }
I20260619 12:02:29.584192 15174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:29.584132 15175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:29.584708 15187 catalog_manager.cc:1259] Loaded cluster ID: bb95aa71a89a42869fd105c06f68c2b5
I20260619 12:02:29.584758 15187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: loading cluster ID for follower catalog manager: success
W20260619 12:02:29.584961 15187 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260619 12:02:29.585006 15187 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260619 12:02:29.585145 15189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:29.585539 15189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:29.585623 15189 catalog_manager.cc:1259] Loaded cluster ID: bb95aa71a89a42869fd105c06f68c2b5
I20260619 12:02:29.585657 15189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:02:29.585813 15189 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 12:02:29.590567 15189 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 12:02:29.593504 15189 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:02:29.593753 15189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Loaded TSK: 0
I20260619 12:02:29.593982 15189 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:02:29.657464 15171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:29.657733 15171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:29.657778 15171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:29.659190 15171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:29.659268 15171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:29.659338 15171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:02:29.661019 15171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:41603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=32813
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:29.661309 15171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:29.661563 15171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:29.666950 15197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.667104 15199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:29.666994 15171 server_base.cc:1034] running on GCE node
W20260619 12:02:29.666942 15196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:29.667451 15171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:29.667774 15171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:29.668959 15171 hybrid_clock.cc:648] HybridClock initialized: now 1781870549668926 us; error 51 us; skew 500 ppm
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:29.671928 15171 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:02:29.672437 15171 webserver.cc:460] Webserver started at http://127.0.203.193:32813/ using document root <none> and password file <none>
I20260619 12:02:29.672767 15171 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:29.672855 15171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:29.674448 15171 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:29.675365 15206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.675645 15171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:29.675776 15171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "01080459e55543dcad28d57fd5048cf6"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "03b84f95831b76ac186643ab9704756d"
server_key_iv: "15d58774682b1c3fb4dc031556b81d2c"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.676250 15171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:29.702360 15171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:29.705044 15171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:29.705235 15171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:29.705488 15171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:29.705937 15213 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:02:29.707818 15171 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:02:29.707907 15171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 12:02:29.707947 15171 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:02:29.708920 15171 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:02:29.708971 15171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:29.709146 15213 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Bootstrap starting.
I20260619 12:02:29.717710 15171 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:41603
I20260619 12:02:29.718325 15171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:02:29.718945 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15171
I20260619 12:02:29.719278 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:33787
--local_ip_for_outbound_sockets=127.0.203.194
--tserver_master_addrs=127.0.203.254:32777
--webserver_port=35203
--webserver_interface=127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:02:29.721259 15213 log.cc:824] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:29.721801 15320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:41603 every 8 connection(s)
I20260619 12:02:29.724378 15213 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:02:29.724792 15213 tablet_bootstrap.cc:492] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Bootstrap complete.
I20260619 12:02:29.726353 15213 ts_tablet_manager.cc:1397] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.010s
I20260619 12:02:29.728529 15213 raft_consensus.cc:348] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.729035 15213 raft_consensus.cc:729] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01080459e55543dcad28d57fd5048cf6, State: Initialized, Role: FOLLOWER
I20260619 12:02:29.729209 15213 consensus_queue.cc:260] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:29.729590 15213 ts_tablet_manager.cc:1428] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:29.736351 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.719241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:56101)
0619 12:02:29.722027 (+ 2786us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.722073 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.722937 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.724386 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.724391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.724986 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:29.725317 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.725324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.727047 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.727049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.728159 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.728162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.729194 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.729195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.729519 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.730314 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:29.730324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:29.731835 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:29.734213 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.734223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.734225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.734486 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.734799 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.734800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.734801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.734927 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.735630 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:29.735637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:29.735801 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.735923 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.736125 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2736,"thread_start_us":95,"threads_started":1}
I20260619 12:02:29.736349 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.721000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:56101 (local address 127.0.203.254:32777)
0619 12:02:29.723874 (+ 2874us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.723885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.723933 (+ 48us) server_negotiation.cc:407] Connection header received
0619 12:02:29.724082 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.724088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.724163 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.724331 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:29.725570 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.726929 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.728264 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.728926 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.729408 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.731975 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:29.731980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:29.731980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:29.731994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:29.734076 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.734610 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.734614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.734614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.734719 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.735002 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.735003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.735003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.735540 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:29.735654 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.735977 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.736125 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2807,"thread_start_us":88,"threads_started":1}
I20260619 12:02:29.737084 15321 heartbeater.cc:347] Connected to a master server at 127.0.203.254:32777
I20260619 12:02:29.737183 15321 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:29.737564 15321 heartbeater.cc:511] Master 127.0.203.254:32777 requested a full tablet report, sending...
I20260619 12:02:29.738423 15133 ts_manager.cc:194] Registered new tserver with Master: 01080459e55543dcad28d57fd5048cf6 (127.0.203.193:41603)
I20260619 12:02:29.739575 15133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:56101
I20260619 12:02:29.740535 15321 heartbeater.cc:503] Master 127.0.203.254:32777 was elected leader, sending a full tablet report...
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 12:02:29.749322 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.739074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41603 (local address 127.0.0.1:44650)
0619 12:02:29.739275 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.739325 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.739403 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.740773 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.740781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.741132 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:29.741197 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.741199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.742096 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.742097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.743059 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.743062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.743559 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.743560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.743650 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.744160 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:29.744168 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:29.745819 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:29.748004 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.748013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.748014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.748243 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.748477 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.748478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.748479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.748548 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.748901 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:29.748908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:29.748967 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.749062 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.749204 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260619 12:02:29.749315 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.739146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.0.203.193:41603)
0619 12:02:29.739939 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.739944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.739955 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:02:29.739998 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.740000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.740013 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.740102 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:29.741290 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.742014 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.743156 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.743477 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.743545 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.745922 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:29.745932 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:29.745933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:29.745946 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:29.747888 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.748344 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.748348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.748348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.748409 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.748610 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.748610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.748611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.748805 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:29.748918 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.749096 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.749203 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":105,"threads_started":1}
I20260619 12:02:29.749755 15255 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d29bbacfab64105ac68e7275e295427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:29 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44650
I20260619 12:02:29.750396 15330 tablet_replica.cc:330] stopping tablet replica
I20260619 12:02:29.750521 15330 raft_consensus.cc:2229] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:29.750609 15330 raft_consensus.cc:2258] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:29.751103 15330 ts_tablet_manager.cc:1905] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:29.753158 15330 ts_tablet_manager.cc:1918] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:29.753257 15330 log.cc:1196] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/6d29bbacfab64105ac68e7275e295427
I20260619 12:02:29.753479 15330 ts_tablet_manager.cc:1939] T 6d29bbacfab64105ac68e7275e295427 P 01080459e55543dcad28d57fd5048cf6: Deleting consensus metadata
I20260619 12:02:29.754072 15119 catalog_manager.cc:4841] TS 01080459e55543dcad28d57fd5048cf6 (127.0.203.193:41603): tablet 6d29bbacfab64105ac68e7275e295427 (table test-table [id=85060eb8828c4c06a20db6cdb2d6cb2e]) successfully deleted
W20260619 12:02:29.815361 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:29.815605 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:29.815635 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:29.817364 15324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:29.817427 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:29.817459 15324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:02:29.819146 15324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:33787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=35203
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:29.819432 15324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:29.819718 15324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:29.825733 15338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.825685 15336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.825788 15335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:29.826203 15324 server_base.cc:1034] running on GCE node
I20260619 12:02:29.826510 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:29.826951 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:29.828155 15324 hybrid_clock.cc:648] HybridClock initialized: now 1781870549828115 us; error 67 us; skew 500 ppm
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:29.831203 15324 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:02:29.831775 15324 webserver.cc:460] Webserver started at http://127.0.203.194:35203/ using document root <none> and password file <none>
I20260619 12:02:29.832024 15324 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:29.832105 15324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:29.833678 15324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:29.834476 15345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.834754 15324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:29.834853 15324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "e28590f0a1334f5087209bb738de2bf4"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "fdfbe191f790e68d8d4b08278f3364ef"
server_key_iv: "4c6dbed50301ff1a3dddd007ddd8348c"
server_key_version: "encryptionkey@0"
I20260619 12:02:29.835292 15324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:29.868949 15324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:29.871596 15324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:29.871819 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:29.872081 15324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:29.872411 15324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:29.872465 15324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.872504 15324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:29.872546 15324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:29.880475 15324 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:33787
I20260619 12:02:29.880542 15458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:33787 every 8 connection(s)
I20260619 12:02:29.881013 15324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:02:29.887984 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15324
I20260619 12:02:29.888391 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:37117
--local_ip_for_outbound_sockets=127.0.203.195
--tserver_master_addrs=127.0.203.254:32777
--webserver_port=38973
--webserver_interface=127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:02:29 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:29.892908 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.881999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:40839 (local address 127.0.203.254:32777)
0619 12:02:29.882102 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:29.882106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:29.882395 (+ 289us) server_negotiation.cc:407] Connection header received
0619 12:02:29.883162 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:29.883164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:29.883184 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:29.883264 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:29.884543 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.885232 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.886357 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.886601 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.886671 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.889289 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:29.889291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:29.889291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:29.889302 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:29.891103 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.891676 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.891678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.891678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.891743 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:29.892071 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:29.892073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:29.892073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:29.892202 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:29.892278 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:29.892729 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 12:02:29.892837 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:02:29.893226 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:29.881967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:40839)
0619 12:02:29.882178 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:29.882228 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:29.883044 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:29.883354 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:29.883359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:29.884146 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:29.884401 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.884406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.885356 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.885357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.886255 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:29.886257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:29.886690 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:29.886691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:29.886821 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:29.887437 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:29.887447 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:29.889177 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:29.891211 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.891217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.891220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.891589 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.891820 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:29.891821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:29.891822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:29.891970 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:29.892290 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:29.892297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:29.892483 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 12:02:29.892616 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 12:02:29.892863 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260619 12:02:29.893909 15459 heartbeater.cc:347] Connected to a master server at 127.0.203.254:32777
I20260619 12:02:29.894076 15459 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:29.894375 15459 heartbeater.cc:511] Master 127.0.203.254:32777 requested a full tablet report, sending...
I20260619 12:02:29.894963 15133 ts_manager.cc:194] Registered new tserver with Master: e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787)
I20260619 12:02:29.895701 15133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:40839
W20260619 12:02:29.984329 15462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:29.984628 15462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:29.984688 15462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:29.987115 15462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:29.987187 15462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:29.987228 15462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:02:29.989490 15462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:37117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=38973
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:29.989789 15462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:29.990078 15462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:29.997805 15467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:29.999607 15468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:29.999585 15462 server_base.cc:1034] running on GCE node
W20260619 12:02:29.999922 15470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:30.000715 15462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:30.001575 15462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:30.002847 15462 hybrid_clock.cc:648] HybridClock initialized: now 1781870550002723 us; error 145 us; skew 500 ppm
Jun 19 12:02:30 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:30.007175 15462 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:02:30.007892 15462 webserver.cc:460] Webserver started at http://127.0.203.195:38973/ using document root <none> and password file <none>
I20260619 12:02:30.008342 15462 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:30.008574 15462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:30.010530 15462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.011662 15477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.012174 15462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.012467 15462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "20f3b0b5e57344d78ef2d16ee15ae343"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "f1125ecb17560a0435ade5c690b029e0"
server_key_iv: "e9a966370195af1d7d7b7e37bc629741"
server_key_version: "encryptionkey@0"
I20260619 12:02:30.012949 15462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:30.037184 15462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:30.040247 15462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:30.040452 15462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:30.040755 15462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:30.041132 15462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:30.041172 15462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.041219 15462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:30.041250 15462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.048957 15462 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:37117
I20260619 12:02:30.049093 15590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:37117 every 8 connection(s)
I20260619 12:02:30.049443 15462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:02:30 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:30.057693 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15462
I20260619 12:02:30.061503 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.050288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:45641 (local address 127.0.203.254:32777)
0619 12:02:30.050379 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.050382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.050668 (+ 286us) server_negotiation.cc:407] Connection header received
0619 12:02:30.051268 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.051269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.051281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.051354 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:30.052508 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.053125 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.054498 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.054714 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.054813 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.057586 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:30.057587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:30.057588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:30.057597 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:30.059735 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:30.060260 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:30.060261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:30.060261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:30.060328 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:30.060730 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:30.060731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:30.060731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:30.060859 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:30.060931 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.061277 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.061372 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:02:30.061658 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.050262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:45641)
0619 12:02:30.050513 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.050557 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:30.051140 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:30.051460 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:30.051464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:30.052089 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:30.052318 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.052323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.053279 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.053280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.054395 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.054399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.054895 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.054897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.055030 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.055628 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:30.055638 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:30.057430 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:30.059848 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:30.059855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:30.059858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:30.060170 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:30.060401 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:30.060401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:30.060402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:30.060547 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:30.060942 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:30.060948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:30.061125 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 12:02:30.061236 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 12:02:30.061389 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":155,"threads_started":1}
I20260619 12:02:30.062309 15591 heartbeater.cc:347] Connected to a master server at 127.0.203.254:32777
I20260619 12:02:30.062449 15591 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:30.062675 15591 heartbeater.cc:511] Master 127.0.203.254:32777 requested a full tablet report, sending...
I20260619 12:02:30.063144 15133 ts_manager.cc:194] Registered new tserver with Master: 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
I20260619 12:02:30.063632 15133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:45641
I20260619 12:02:30.066220 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.058141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.0.203.254:32777)
0619 12:02:30.058266 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.058268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.058370 (+ 102us) server_negotiation.cc:407] Connection header received
0619 12:02:30.058442 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.058443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.058457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.058531 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:30.059266 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.060041 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.061125 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.061395 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.061485 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.062958 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:30.062959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:30.062960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:30.062969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:30.064835 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:30.065245 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:30.065246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:30.065247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:30.065318 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:30.065629 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:30.065630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:30.065630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:30.065758 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:30.065946 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.066056 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.066139 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260619 12:02:30.067384 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:02:30.070542 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.067882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.0.203.254:32777)
0619 12:02:30.067930 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.067932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.068030 (+ 98us) server_negotiation.cc:407] Connection header received
0619 12:02:30.068084 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.068085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.068114 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.068196 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:02:30.068414 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.068987 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.069810 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.070010 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.070083 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.070228 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:02:30.070327 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:02:30.070391 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.070462 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.070481 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:02:30.070919 15133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:02:30.072561 15133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:02:30.075258 15255 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa561573c2a44e08f1d24f542cb97aa (DEFAULT_TABLE table=test-table [id=26c0309545ef4068a0e7ecec3b6d242f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:30.075525 15255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa561573c2a44e08f1d24f542cb97aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:30.077471 15213 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Bootstrap starting.
I20260619 12:02:30.077968 15213 tablet_bootstrap.cc:654] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:30.078794 15213 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: No bootstrap required, opened a new log
I20260619 12:02:30.078859 15213 ts_tablet_manager.cc:1397] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.079245 15213 raft_consensus.cc:348] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.079352 15213 raft_consensus.cc:374] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:30.079437 15213 raft_consensus.cc:729] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01080459e55543dcad28d57fd5048cf6, State: Initialized, Role: FOLLOWER
I20260619 12:02:30.079504 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.074932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:33787 (local address 127.0.0.1:49532)
0619 12:02:30.074985 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.074998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:30.075035 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:30.075695 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:30.075696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:30.075698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:30.075752 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.075754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.077063 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.077064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.078112 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.078116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.079194 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.079195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.079274 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.079299 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:02:30.079370 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:02:30.079428 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 12:02:30.079488 15213 consensus_queue.cc:260] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.079551 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.075149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.0.203.194:33787)
0619 12:02:30.075488 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.075494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.075508 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:02:30.075559 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.075562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.075582 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.075686 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:30.075838 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.076943 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.078267 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.079098 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.079171 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.079233 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.079403 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.079448 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":167,"threads_started":1}
I20260619 12:02:30.079603 15213 ts_tablet_manager.cc:1428] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.080713 15393 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa561573c2a44e08f1d24f542cb97aa (DEFAULT_TABLE table=test-table [id=26c0309545ef4068a0e7ecec3b6d242f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:30.081085 15393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa561573c2a44e08f1d24f542cb97aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:30.081426 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.074964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:37117 (local address 127.0.0.1:32802)
0619 12:02:30.075420 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.075432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:30.075486 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:30.077786 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:30.077788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:30.077792 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:30.077855 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.077857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.079113 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.079114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.080148 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.080150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.081160 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.081161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.081225 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.081230 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:02:30.081288 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 12:02:30.081339 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":68,"threads_started":1}
I20260619 12:02:30.081444 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.075200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.0.203.195:37117)
0619 12:02:30.077606 (+ 2406us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.077610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.077620 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:02:30.077666 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.077668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.077681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.077772 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:30.077909 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.078973 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.080308 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.081060 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.081153 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.081247 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.081304 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.081346 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2365,"thread_start_us":120,"threads_started":1}
I20260619 12:02:30.082307 15525 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa561573c2a44e08f1d24f542cb97aa (DEFAULT_TABLE table=test-table [id=26c0309545ef4068a0e7ecec3b6d242f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:30.082655 15525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa561573c2a44e08f1d24f542cb97aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:30.084390 15601 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Bootstrap starting.
I20260619 12:02:30.085067 15601 tablet_bootstrap.cc:654] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:30.085624 15601 log.cc:824] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:30.086326 15603 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Bootstrap starting.
I20260619 12:02:30.086591 15601 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: No bootstrap required, opened a new log
I20260619 12:02:30.086683 15601 ts_tablet_manager.cc:1397] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:30.086879 15603 tablet_bootstrap.cc:654] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:30.087291 15603 log.cc:824] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:30.088110 15603 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: No bootstrap required, opened a new log
I20260619 12:02:30.088189 15603 ts_tablet_manager.cc:1397] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:02:30.088948 15601 raft_consensus.cc:348] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.089139 15601 raft_consensus.cc:374] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:30.089180 15601 raft_consensus.cc:729] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28590f0a1334f5087209bb738de2bf4, State: Initialized, Role: FOLLOWER
I20260619 12:02:30.089330 15601 consensus_queue.cc:260] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.089676 15601 ts_tablet_manager.cc:1428] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:02:30.089766 15459 heartbeater.cc:503] Master 127.0.203.254:32777 was elected leader, sending a full tablet report...
I20260619 12:02:30.090292 15603 raft_consensus.cc:348] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.090507 15603 raft_consensus.cc:374] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:30.090549 15603 raft_consensus.cc:729] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f3b0b5e57344d78ef2d16ee15ae343, State: Initialized, Role: FOLLOWER
I20260619 12:02:30.090698 15603 consensus_queue.cc:260] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.091041 15603 ts_tablet_manager.cc:1428] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:02:30.091082 15591 heartbeater.cc:503] Master 127.0.203.254:32777 was elected leader, sending a full tablet report...
W20260619 12:02:30.131759 15460 tablet.cc:2369] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:30.181372 15327 raft_consensus.cc:482] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:30.181515 15327 raft_consensus.cc:504] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.182089 15327 leader_election.cc:288] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
I20260619 12:02:30.186841 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.182430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:59667 (local address 127.0.203.194:33787)
0619 12:02:30.182507 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.182510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.182532 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:02:30.182568 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.182569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.182579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.182638 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:30.182915 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.183723 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.185671 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.186323 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.186424 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.186436 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.186692 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.186754 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:02:30.186841 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.182402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:37615 (local address 127.0.203.195:37117)
0619 12:02:30.182469 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.182471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.182622 (+ 151us) server_negotiation.cc:407] Connection header received
0619 12:02:30.182719 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.182721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.182731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.182777 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:30.182942 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.183757 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.185671 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.186324 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.186424 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.186436 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.186692 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.186751 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 12:02:30.186844 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.182326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:33787 (local address 127.0.203.193:59667)
0619 12:02:30.182379 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.182394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:30.182481 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:30.182672 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:30.182674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:30.182676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:30.182817 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.182819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.183868 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.183870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.185526 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.185529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.186453 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.186455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.186581 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.186584 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:02:30.186666 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:02:30.186723 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":148}
I20260619 12:02:30.186852 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.182356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:37117 (local address 127.0.203.193:37615)
0619 12:02:30.182537 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.182547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:30.182641 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:30.182800 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:30.182800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:30.182802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:30.182861 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.182870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.183868 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.183870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.185526 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:30.185529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.186474 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.186476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:30.186546 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.186577 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 12:02:30.186666 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:02:30.186723 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":103,"thread_start_us":100,"threads_started":1}
I20260619 12:02:30.187402 15545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa561573c2a44e08f1d24f542cb97aa" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" is_pre_election: true
I20260619 12:02:30.187418 15413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa561573c2a44e08f1d24f542cb97aa" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28590f0a1334f5087209bb738de2bf4" is_pre_election: true
I20260619 12:02:30.187597 15545 raft_consensus.cc:2454] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 0.
I20260619 12:02:30.187618 15413 raft_consensus.cc:2454] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 0.
I20260619 12:02:30.187889 15208 leader_election.cc:302] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6, 20f3b0b5e57344d78ef2d16ee15ae343; no voters:
I20260619 12:02:30.188050 15327 raft_consensus.cc:2781] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:02:30.188114 15327 raft_consensus.cc:482] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:02:30.188151 15327 raft_consensus.cc:3037] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:30.189065 15327 raft_consensus.cc:504] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.189234 15327 leader_election.cc:288] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 election: Requested vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
I20260619 12:02:30.189450 15413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa561573c2a44e08f1d24f542cb97aa" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28590f0a1334f5087209bb738de2bf4"
I20260619 12:02:30.189455 15545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa561573c2a44e08f1d24f542cb97aa" candidate_uuid: "01080459e55543dcad28d57fd5048cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f3b0b5e57344d78ef2d16ee15ae343"
I20260619 12:02:30.189548 15413 raft_consensus.cc:3037] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:30.189549 15545 raft_consensus.cc:3037] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:30.190362 15413 raft_consensus.cc:2454] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 1.
I20260619 12:02:30.190483 15545 raft_consensus.cc:2454] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01080459e55543dcad28d57fd5048cf6 in term 1.
I20260619 12:02:30.190644 15210 leader_election.cc:302] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6, e28590f0a1334f5087209bb738de2bf4; no voters:
I20260619 12:02:30.190798 15327 raft_consensus.cc:2781] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:30.191045 15327 raft_consensus.cc:686] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 LEADER]: Becoming Leader. State: Replica: 01080459e55543dcad28d57fd5048cf6, State: Running, Role: LEADER
I20260619 12:02:30.191156 15327 consensus_queue.cc:237] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.192152 15133 catalog_manager.cc:5495] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01080459e55543dcad28d57fd5048cf6 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "01080459e55543dcad28d57fd5048cf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } health_report { overall_health: HEALTHY } } }
I20260619 12:02:30.199476 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.196955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.0.203.193:41603)
0619 12:02:30.197040 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:30.197042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:30.197049 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:02:30.197080 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:30.197080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:30.197097 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:30.197177 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:02:30.197316 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.197857 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.198656 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:30.198872 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:30.198960 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:30.199122 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:02:30.199236 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:02:30.199316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:30.199354 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 12:02:30.199381 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:02:30.201582 15413 raft_consensus.cc:1261] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Refusing update from remote peer 01080459e55543dcad28d57fd5048cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:30.201620 15545 raft_consensus.cc:1261] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Refusing update from remote peer 01080459e55543dcad28d57fd5048cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:30.201980 15608 consensus_queue.cc:1035] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:30.202148 15327 consensus_queue.cc:1035] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:30.205497 15610 mvcc.cc:205] Tried to move back new op lower bound from 7298541773622833152 to 7298541773583896576. Current Snapshot: MvccSnapshot[applied={T|T < 7298541773622833152}]
I20260619 12:02:30.206192 15611 mvcc.cc:205] Tried to move back new op lower bound from 7298541773622833152 to 7298541773583896576. Current Snapshot: MvccSnapshot[applied={T|T < 7298541773622833152}]
I20260619 12:02:30.206478 15612 mvcc.cc:205] Tried to move back new op lower bound from 7298541773622833152 to 7298541773583896576. Current Snapshot: MvccSnapshot[applied={T|T < 7298541773622833152 or (T in {7298541773622833152})}]
I20260619 12:02:30.208153 15133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:30.208257 15133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:30.209267 15133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Sending DeleteTablet for 3 replicas of tablet 8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:30.209719 15255 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa561573c2a44e08f1d24f542cb97aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:30 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:44650
I20260619 12:02:30.209719 15525 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa561573c2a44e08f1d24f542cb97aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:30 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:32802
I20260619 12:02:30.209851 15330 tablet_replica.cc:330] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: stopping tablet replica
I20260619 12:02:30.209832 15393 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa561573c2a44e08f1d24f542cb97aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:30 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:49532
I20260619 12:02:30.209915 15330 raft_consensus.cc:2229] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:02:30.210038 15330 raft_consensus.cc:2258] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:30.210228 15620 tablet_replica.cc:330] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: stopping tablet replica
I20260619 12:02:30.210300 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 15171
I20260619 12:02:30.210332 15620 raft_consensus.cc:2229] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:30.210482 15330 ts_tablet_manager.cc:1905] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:30.211189 15621 tablet_replica.cc:330] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: stopping tablet replica
I20260619 12:02:30.211336 15621 raft_consensus.cc:2229] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:30.211478 15621 raft_consensus.cc:2258] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:30.211656 15620 raft_consensus.cc:2258] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:30.212136 15621 ts_tablet_manager.cc:1905] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:30.212296 15620 ts_tablet_manager.cc:1905] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:30.213958 15621 ts_tablet_manager.cc:1918] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:30.214071 15621 log.cc:1196] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:30.214138 15620 ts_tablet_manager.cc:1918] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:30.214216 15620 log.cc:1196] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:30.214462 15621 ts_tablet_manager.cc:1939] T 8aa561573c2a44e08f1d24f542cb97aa P e28590f0a1334f5087209bb738de2bf4: Deleting consensus metadata
I20260619 12:02:30.214558 15620 ts_tablet_manager.cc:1939] T 8aa561573c2a44e08f1d24f542cb97aa P 20f3b0b5e57344d78ef2d16ee15ae343: Deleting consensus metadata
I20260619 12:02:30.215016 15120 catalog_manager.cc:4841] TS e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787): tablet 8aa561573c2a44e08f1d24f542cb97aa (table test-table [id=26c0309545ef4068a0e7ecec3b6d242f]) successfully deleted
I20260619 12:02:30.215045 15118 catalog_manager.cc:4841] TS 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117): tablet 8aa561573c2a44e08f1d24f542cb97aa (table test-table [id=26c0309545ef4068a0e7ecec3b6d242f]) successfully deleted
W20260619 12:02:30.218048 15119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:41603 (error 108)
I20260619 12:02:30.218161 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 15324
I20260619 12:02:30.219514 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.219313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41603 (local address 127.0.0.1:44666)
0619 12:02:30.219389 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.219458 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:02:30.219714 15119 catalog_manager.cc:4584] TS 01080459e55543dcad28d57fd5048cf6 (127.0.203.193:41603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8aa561573c2a44e08f1d24f542cb97aa: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41603: connect: Connection refused (error 111)
I20260619 12:02:30.224776 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 15462
I20260619 12:02:30.230756 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 15100
I20260619 12:02:30.237636 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:32777
--webserver_interface=127.0.203.254
--webserver_port=46141
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:30.326335 15623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:30.326673 15623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:30.326726 15623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:30.328369 15623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:30.328482 15623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:30.328510 15623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:30.328532 15623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:30.328553 15623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:30.330938 15623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:30.331485 15623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:30.331771 15623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:30.338007 15628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:30.338007 15631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:30.338020 15629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:30.338477 15623 server_base.cc:1034] running on GCE node
I20260619 12:02:30.338783 15623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:30.339116 15623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:30.340288 15623 hybrid_clock.cc:648] HybridClock initialized: now 1781870550340269 us; error 42 us; skew 500 ppm
Jun 19 12:02:30 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:30.343353 15623 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:30.343870 15623 webserver.cc:460] Webserver started at http://127.0.203.254:46141/ using document root <none> and password file <none>
I20260619 12:02:30.344137 15623 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:30.344233 15623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:30.345837 15623 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.346716 15638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.347029 15623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.347152 15623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "9dabfc72a7ac49d182e07b709e4b9af1"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "6fadb05c19f5ed1178aeb91c89fb8e8d"
server_key_iv: "7592e2bebcbbaf5354b005fa4ac1ecdb"
server_key_version: "encryptionkey@0"
I20260619 12:02:30.347620 15623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:30.359781 15623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:30.362356 15623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:30.362488 15623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:30.367470 15623 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:32777
I20260619 12:02:30.367506 15690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:32777 every 8 connection(s)
I20260619 12:02:30.367944 15623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:30.369626 15691 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:02:30.370307 15691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Bootstrap starting.
I20260619 12:02:30.373978 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15623
I20260619 12:02:30.374362 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:41603
--local_ip_for_outbound_sockets=127.0.203.193
--tserver_master_addrs=127.0.203.254:32777
--webserver_port=32813
--webserver_interface=127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:02:30.379422 15691 log.cc:824] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:30.381805 15691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:02:30.382148 15691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: Bootstrap complete.
I20260619 12:02:30.384464 15691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:30.384781 15691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dabfc72a7ac49d182e07b709e4b9af1, State: Initialized, Role: FOLLOWER
I20260619 12:02:30.384984 15691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:30.385078 15691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:02:30.385130 15691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:02:30.385182 15691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:02:30.386210 15691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:30.386332 15691 leader_election.cc:302] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dabfc72a7ac49d182e07b709e4b9af1; no voters:
I20260619 12:02:30.386549 15691 leader_election.cc:288] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 12:02:30.386586 15695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:02:30.386705 15695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [term 3 LEADER]: Becoming Leader. State: Replica: 9dabfc72a7ac49d182e07b709e4b9af1, State: Running, Role: LEADER
I20260619 12:02:30.386799 15695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } }
I20260619 12:02:30.386830 15691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:30.388569 15697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } } }
I20260619 12:02:30.388998 15697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:30.389317 15696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dabfc72a7ac49d182e07b709e4b9af1. Latest consensus state: current_term: 3 leader_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dabfc72a7ac49d182e07b709e4b9af1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 32777 } } }
I20260619 12:02:30.389408 15696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:30.389490 15709 catalog_manager.cc:1259] Loaded cluster ID: bb95aa71a89a42869fd105c06f68c2b5
I20260619 12:02:30.389823 15709 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: loading cluster ID for follower catalog manager: success
W20260619 12:02:30.390064 15709 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9dabfc72a7ac49d182e07b709e4b9af1: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260619 12:02:30.390102 15709 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260619 12:02:30.390159 15712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:30.390615 15712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:30.390808 15712 catalog_manager.cc:1259] Loaded cluster ID: bb95aa71a89a42869fd105c06f68c2b5
I20260619 12:02:30.390877 15712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 12:02:30.391042 15712 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe44a5c49ad google::LogMessage::Fail() at ??:0
@ 0x7fe44a5c788a google::LogMessage::SendToLog() at ??:0
@ 0x7fe44a5c44de google::LogMessage::Flush() at ??:0
@ 0x7fe44a5c5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4524e90f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe4524f28b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe44c4920d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe44c487954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe44d2eb6db start_thread at ??:0
@ 0x7fe4490c261f clone at ??:0
W20260619 12:02:30.479789 15693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:30.480038 15693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:30.480065 15693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:30.481631 15693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:30.481699 15693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:30.481731 15693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:02:30.483436 15693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:41603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=32813
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:30.483717 15693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:30.483973 15693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:30.489518 15727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:30.489549 15730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:30.489748 15693 server_base.cc:1034] running on GCE node
W20260619 12:02:30.489838 15728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:30.490262 15693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:30.490728 15693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:30.491966 15693 hybrid_clock.cc:648] HybridClock initialized: now 1781870550491899 us; error 73 us; skew 500 ppm
Jun 19 12:02:30 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:30.496027 15693 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:02:30.496734 15693 webserver.cc:460] Webserver started at http://127.0.203.193:32813/ using document root <none> and password file <none>
I20260619 12:02:30.497080 15693 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:30.497160 15693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:30.499459 15693 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:02:30.500466 15737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.500758 15693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:30.500880 15693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "01080459e55543dcad28d57fd5048cf6"
format_stamp: "Formatted at 2026-06-19 12:02:28 on dist-test-slave-fbq8"
server_key: "03b84f95831b76ac186643ab9704756d"
server_key_iv: "15d58774682b1c3fb4dc031556b81d2c"
server_key_version: "encryptionkey@0"
I20260619 12:02:30.501401 15693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:30.525157 15693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:30.528215 15693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:30.528398 15693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:30.528745 15693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:30.529351 15744 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:02:30.531855 15693 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:02:30.531946 15693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260619 12:02:30.532001 15693 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:02:30.532948 15693 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:02:30.533010 15693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:30.533169 15744 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Bootstrap starting.
I20260619 12:02:30.535439 15744 log.cc:824] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:30.539556 15744 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:02:30.540019 15744 tablet_bootstrap.cc:492] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Bootstrap complete.
I20260619 12:02:30.540225 15744 ts_tablet_manager.cc:1397] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260619 12:02:30.541745 15693 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:41603
I20260619 12:02:30.542157 15693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:02:30.542712 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:41603 every 8 connection(s)
I20260619 12:02:30.542805 15744 raft_consensus.cc:348] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.543520 15744 raft_consensus.cc:729] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01080459e55543dcad28d57fd5048cf6, State: Initialized, Role: FOLLOWER
I20260619 12:02:30.543713 15744 consensus_queue.cc:260] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:30.544112 15744 ts_tablet_manager.cc:1428] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 12:02:30.551644 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15693
I20260619 12:02:30.551994 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:33787
--local_ip_for_outbound_sockets=127.0.203.194
--tserver_master_addrs=127.0.203.254:32777
--webserver_port=35203
--webserver_interface=127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:30.641547 15855 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:02:30.543026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:51293)
0619 12:02:30.543262 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.543310 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:30.543518 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:30.641291 (+ 97773us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
W20260619 12:02:30.642026 15853 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:32777 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.254:32777: Client connection negotiation failed: client connection to 127.0.203.254:32777: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:02:30.642513 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.642316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:58671)
0619 12:02:30.642406 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.642469 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 12:02:30.643172 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.642959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:37897)
0619 12:02:30.643063 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.643147 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 12:02:30.643350 15853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:02:30.646790 15857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:30.647006 15857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:30.647030 15857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:30.648259 15857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:30.648306 15857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:30.648334 15857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:02:30.649912 15857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:33787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=35203
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:30.650166 15857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:30.650419 15857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:30.656368 15863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:30.656488 15865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:30.656345 15857 server_base.cc:1034] running on GCE node
W20260619 12:02:30.656352 15862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:30.657068 15857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:30.657346 15857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:30.658528 15857 hybrid_clock.cc:648] HybridClock initialized: now 1781870550658519 us; error 40 us; skew 500 ppm
Jun 19 12:02:30 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:30.661641 15857 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:02:30.662159 15857 webserver.cc:460] Webserver started at http://127.0.203.194:35203/ using document root <none> and password file <none>
I20260619 12:02:30.662393 15857 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:30.662448 15857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:30.664170 15857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:02:30.665169 15872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.665472 15857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:02:30.665585 15857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "e28590f0a1334f5087209bb738de2bf4"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "fdfbe191f790e68d8d4b08278f3364ef"
server_key_iv: "4c6dbed50301ff1a3dddd007ddd8348c"
server_key_version: "encryptionkey@0"
I20260619 12:02:30.665988 15857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:30.679618 15857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:30.682082 15857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:30.682256 15857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:30.682521 15857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:30.682853 15857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:30.682905 15857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.682945 15857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:30.682994 15857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.690779 15857 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:33787
I20260619 12:02:30.690836 15985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:33787 every 8 connection(s)
I20260619 12:02:30.691202 15857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:02:30.692407 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.691950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:36605)
0619 12:02:30.692126 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.692233 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
W20260619 12:02:30.692978 15986 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:32777 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.254:32777: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
I20260619 12:02:30.693356 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.693228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:47875)
0619 12:02:30.693305 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.693337 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 12:02:30.693713 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.693628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:48255)
0619 12:02:30.693670 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.693695 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 12:02:30.693847 15986 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:02:30.698071 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15857
I20260619 12:02:30.698357 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:37117
--local_ip_for_outbound_sockets=127.0.203.195
--tserver_master_addrs=127.0.203.254:32777
--webserver_port=38973
--webserver_interface=127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:30.786269 15989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:30.786528 15989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:30.786566 15989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:30.787730 15989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:30.787807 15989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:30.787851 15989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:02:30.789418 15989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:37117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=38973
--tserver_master_addrs=127.0.203.254:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:30.789701 15989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:30.789945 15989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:30.796001 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:30.796072 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:30.796165 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:30.796238 15989 server_base.cc:1034] running on GCE node
I20260619 12:02:30.796594 15989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:30.796904 15989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:30.798097 15989 hybrid_clock.cc:648] HybridClock initialized: now 1781870550798085 us; error 39 us; skew 500 ppm
Jun 19 12:02:30 dist-test-slave-fbq8 krb5kdc[14552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:30.800916 15989 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:02:30.801407 15989 webserver.cc:460] Webserver started at http://127.0.203.195:38973/ using document root <none> and password file <none>
I20260619 12:02:30.801685 15989 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:30.801738 15989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:30.803412 15989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.804286 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.804536 15989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:30.804620 15989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "20f3b0b5e57344d78ef2d16ee15ae343"
format_stamp: "Formatted at 2026-06-19 12:02:29 on dist-test-slave-fbq8"
server_key: "f1125ecb17560a0435ade5c690b029e0"
server_key_iv: "e9a966370195af1d7d7b7e37bc629741"
server_key_version: "encryptionkey@0"
I20260619 12:02:30.805112 15989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:30.822728 15989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:30.825467 15989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:30.825603 15989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:30.825953 15989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:30.826334 15989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:30.826393 15989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.826438 15989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:30.826467 15989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:30.834331 15989 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:37117
I20260619 12:02:30.834403 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:37117 every 8 connection(s)
I20260619 12:02:30.834906 15989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:02:30.836344 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.835867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:38521)
0619 12:02:30.836071 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.836166 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
W20260619 12:02:30.836912 16118 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:32777 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.254:32777: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
I20260619 12:02:30.837282 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.837174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:49987)
0619 12:02:30.837232 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.837263 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 12:02:30.837589 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:30.837516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:55333)
0619 12:02:30.837557 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:30.837577 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 12:02:30.837721 16118 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:02:30.845039 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 15989
I20260619 12:02:31.644227 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:31.643944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:58499)
0619 12:02:31.644110 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:31.644163 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260619 12:02:31.694610 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:31.694318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:60481)
0619 12:02:31.694491 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:31.694548 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260619 12:02:31.838387 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:31.838123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:59783)
0619 12:02:31.838280 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:31.838322 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260619 12:02:32.411531 16125 raft_consensus.cc:482] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:32.411677 16125 raft_consensus.cc:504] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:32.412163 16125 leader_election.cc:288] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
Jun 19 12:02:32 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.195@KRBTEST.COM
Jun 19 12:02:32 dist-test-slave-fbq8 krb5kdc[14552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.194@KRBTEST.COM
I20260619 12:02:32.424268 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:32.412396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:37117 (local address 127.0.203.193:36303)
0619 12:02:32.412554 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:32.412568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:32.412666 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:32.413542 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:32.413546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:32.413880 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:32.414269 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:32.414271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.415382 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.415382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:32.416217 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:32.416220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.416989 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.416990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:32.417149 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:32.417951 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:32.417961 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:32.419829 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:32.422264 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:32.422272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:32.422275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:32.422584 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:32.422938 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:32.422939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:32.422939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:32.423087 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:32.423610 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:32.423617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:32.423763 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 12:02:32.423891 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 12:02:32.424123 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":35,"thread_start_us":61,"threads_started":1}
I20260619 12:02:32.424288 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:32.412555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:36303 (local address 127.0.203.195:37117)
0619 12:02:32.412777 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:32.412787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:32.412807 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:02:32.413289 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:32.413291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:32.413339 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:32.413507 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:32.414392 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.415260 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.416312 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.416873 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.416999 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:32.420025 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:32.420032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:32.420033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:32.420050 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:32.422143 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:32.422700 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:32.422704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:32.422704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:32.422834 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:32.423190 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:32.423191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:32.423191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:32.423519 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:32.423591 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:32.423980 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 12:02:32.424146 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":63,"threads_started":1}
I20260619 12:02:32.424407 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:32.412534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:38157 (local address 127.0.203.194:33787)
0619 12:02:32.412777 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:32.412786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:32.412807 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:02:32.413423 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:32.413426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:32.413468 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:32.413627 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:32.414253 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.415110 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.416449 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.417318 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.417436 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:32.420315 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:32.420324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:32.420325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:32.420336 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:32.422365 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:32.422947 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:32.422951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:32.422951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:32.423067 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:32.423376 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:32.423377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:32.423378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:32.423761 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:32.423910 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:32.424142 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 12:02:32.424283 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
I20260619 12:02:32.424798 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:32.412396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:33787 (local address 127.0.203.193:38157)
0619 12:02:32.412533 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:32.412546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:32.412610 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:32.413612 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:32.413613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:32.413907 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:32.414127 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:32.414134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.415417 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.415419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:32.416287 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:32.416291 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:32.417455 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:32.417457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:32.417558 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:32.418057 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:32.418061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:32.420196 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:32.422482 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:32.422483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:32.422484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:32.422779 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:32.423191 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:32.423192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:32.423192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:32.423265 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:32.423904 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:32.423906 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:32.423971 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 12:02:32.424062 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 12:02:32.424254 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":91,"thread_start_us":69,"threads_started":1}
W20260619 12:02:32.425495 15739 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
W20260619 12:02:32.425978 15741 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:32.426038 15741 leader_election.cc:302] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6; no voters: 20f3b0b5e57344d78ef2d16ee15ae343, e28590f0a1334f5087209bb738de2bf4
I20260619 12:02:32.426170 16125 raft_consensus.cc:2726] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:02:32.645053 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:32.644878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:47539)
0619 12:02:32.644945 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:32.644998 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 12:02:32.695644 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:32.695324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:44893)
0619 12:02:32.695512 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:32.695577 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":122,"threads_started":1}
I20260619 12:02:32.839322 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:32.839011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:38627)
0619 12:02:32.839209 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:32.839275 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260619 12:02:33.645995 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:33.645699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:54483)
0619 12:02:33.645876 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:33.645932 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260619 12:02:33.696558 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:33.696271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:60623)
0619 12:02:33.696453 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:33.696508 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260619 12:02:33.840289 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:33.839879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:46455)
0619 12:02:33.840049 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:33.840093 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260619 12:02:34.379897 16135 raft_consensus.cc:482] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:34.380020 16135 raft_consensus.cc:504] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:34.380252 16135 leader_election.cc:288] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
W20260619 12:02:34.380738 15741 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
W20260619 12:02:34.380853 15739 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:34.381009 15739 leader_election.cc:302] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6; no voters: 20f3b0b5e57344d78ef2d16ee15ae343, e28590f0a1334f5087209bb738de2bf4
I20260619 12:02:34.381093 16135 raft_consensus.cc:2726] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:02:34.647053 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:34.646716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:34109)
0619 12:02:34.646935 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:34.646999 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
I20260619 12:02:34.697708 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:34.697298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:39217)
0619 12:02:34.697579 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:34.697634 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":212,"threads_started":1}
I20260619 12:02:34.841259 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:34.840945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:44039)
0619 12:02:34.841123 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:34.841175 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260619 12:02:35.648149 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:35.647821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:44685)
0619 12:02:35.648038 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:35.648085 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":123,"threads_started":1}
I20260619 12:02:35.698786 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:35.698411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:45817)
0619 12:02:35.698674 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:35.698733 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":128,"threads_started":1}
I20260619 12:02:35.842229 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:35.841962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:37483)
0619 12:02:35.842107 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:35.842178 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260619 12:02:36.176899 16142 raft_consensus.cc:482] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:36.177057 16142 raft_consensus.cc:504] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:36.177265 16142 leader_election.cc:288] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
W20260619 12:02:36.177914 15741 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
W20260619 12:02:36.177991 15739 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:36.178023 15741 leader_election.cc:302] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6; no voters: 20f3b0b5e57344d78ef2d16ee15ae343, e28590f0a1334f5087209bb738de2bf4
I20260619 12:02:36.178107 16142 raft_consensus.cc:2726] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:02:36.649098 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:36.648839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:40075)
0619 12:02:36.648991 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:36.649034 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260619 12:02:36.699855 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:36.699504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:59833)
0619 12:02:36.699738 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:36.699795 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
I20260619 12:02:36.843174 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:36.842860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:47081)
0619 12:02:36.843049 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:36.843111 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260619 12:02:37.649959 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:37.649645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:40061)
0619 12:02:37.649830 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:37.649891 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260619 12:02:37.700790 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:37.700426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:33933)
0619 12:02:37.700618 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:37.700739 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260619 12:02:37.844255 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:37.843857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:38079)
0619 12:02:37.844120 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:37.844187 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
I20260619 12:02:38.651185 16149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:38.650691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:46639)
0619 12:02:38.650933 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:38.651003 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":94,"threads_started":1}
I20260619 12:02:38.701745 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:38.701397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:37419)
0619 12:02:38.701630 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:38.701692 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260619 12:02:38.845453 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:38.845137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:47995)
0619 12:02:38.845338 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:38.845399 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260619 12:02:39.652451 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:39.652109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:54963)
0619 12:02:39.652332 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:39.652390 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260619 12:02:39.702761 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:39.702468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:38839)
0619 12:02:39.702646 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:39.702697 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260619 12:02:39.846550 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:39.846178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:34675)
0619 12:02:39.846423 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:39.846486 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20260619 12:02:40.653501 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:40.653151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:36369)
0619 12:02:40.653384 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:40.653446 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":114,"threads_started":1}
I20260619 12:02:40.703622 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:40.703373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:46859)
0619 12:02:40.703533 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:40.703572 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260619 12:02:40.847604 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:40.847284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:34391)
0619 12:02:40.847493 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:40.847552 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
I20260619 12:02:41.159797 16158 raft_consensus.cc:482] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:41.159950 16158 raft_consensus.cc:504] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:41.160282 16158 leader_election.cc:288] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
W20260619 12:02:41.160916 15741 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
W20260619 12:02:41.161104 15739 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:41.161209 15739 leader_election.cc:302] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6; no voters: 20f3b0b5e57344d78ef2d16ee15ae343, e28590f0a1334f5087209bb738de2bf4
I20260619 12:02:41.161298 16158 raft_consensus.cc:2726] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:02:41.654754 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:41.654412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:34247)
0619 12:02:41.654605 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:41.654674 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
I20260619 12:02:41.704622 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:41.704289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:44637)
0619 12:02:41.704508 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:41.704570 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":119,"threads_started":1}
I20260619 12:02:41.848728 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:41.848284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:43169)
0619 12:02:41.848556 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:41.848633 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":148,"threads_started":1}
I20260619 12:02:42.655798 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:42.655479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:37339)
0619 12:02:42.655652 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:42.655730 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260619 12:02:42.705580 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:42.705235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:45429)
0619 12:02:42.705471 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:42.705523 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":66,"threads_started":1}
I20260619 12:02:42.849635 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:42.849347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:38231)
0619 12:02:42.849546 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:42.849586 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":126,"threads_started":1}
I20260619 12:02:43.656832 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:43.656451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:51743)
0619 12:02:43.656635 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:43.656778 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260619 12:02:43.706635 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:43.706359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:57905)
0619 12:02:43.706535 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:43.706572 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260619 12:02:43.850695 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:43.850398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:39771)
0619 12:02:43.850578 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:43.850643 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260619 12:02:44.657969 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:44.657666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:53439)
0619 12:02:44.657846 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:44.657918 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260619 12:02:44.707417 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:44.707113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:55071)
0619 12:02:44.707314 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:44.707367 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":99,"threads_started":1}
I20260619 12:02:44.846962 16170 raft_consensus.cc:482] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:44.847095 16170 raft_consensus.cc:504] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e28590f0a1334f5087209bb738de2bf4" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 33787 } } peers { permanent_uuid: "20f3b0b5e57344d78ef2d16ee15ae343" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 37117 } } peers { permanent_uuid: "01080459e55543dcad28d57fd5048cf6" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41603 } }
I20260619 12:02:44.847323 16170 leader_election.cc:288] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787), 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117)
W20260619 12:02:44.847867 15741 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e28590f0a1334f5087209bb738de2bf4 (127.0.203.194:33787): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
W20260619 12:02:44.848024 15739 leader_election.cc:341] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f3b0b5e57344d78ef2d16ee15ae343 (127.0.203.195:37117): Not found: Tablet not found: 8aa561573c2a44e08f1d24f542cb97aa
I20260619 12:02:44.848097 15739 leader_election.cc:302] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01080459e55543dcad28d57fd5048cf6; no voters: 20f3b0b5e57344d78ef2d16ee15ae343, e28590f0a1334f5087209bb738de2bf4
I20260619 12:02:44.848224 16170 raft_consensus.cc:2726] T 8aa561573c2a44e08f1d24f542cb97aa P 01080459e55543dcad28d57fd5048cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:02:44.851573 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:44.851283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:37761)
0619 12:02:44.851461 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:44.851517 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260619 12:02:45.659068 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:45.658747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.193:45447)
0619 12:02:45.658931 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:45.658990 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260619 12:02:45.708501 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:45.708213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.194:39475)
0619 12:02:45.708376 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:45.708442 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260619 12:02:45.848528 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 15693
I20260619 12:02:45.852524 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:45.852292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:32777 (local address 127.0.203.195:53937)
0619 12:02:45.852436 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:45.852474 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260619 12:02:45.854568 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 15857
I20260619 12:02:45.859800 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 15989
2026-06-19T12:02:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18536 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:02:45 dist-test-slave-fbq8 krb5kdc[16177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:02:45 dist-test-slave-fbq8 krb5kdc[16177](info): set up 2 sockets
Jun 19 12:02:45 dist-test-slave-fbq8 krb5kdc[16177](info): commencing operation
krb5kdc: starting...
W20260619 12:02:46.929107 815 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:02:47.071585 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44921
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:47.155977 16190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:47.156270 16190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:47.156311 16190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:47.157652 16190 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:02:47.157723 16190 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:02:47.157744 16190 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:02:47.157763 16190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:02:47.157792 16190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:47.157824 16190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:47.157850 16190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:02:47.157876 16190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:02:47.159444 16190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:47.159729 16190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:47.159976 16190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:47.165666 16198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:47.165644 16195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:47.165650 16196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:47.165648 16190 server_base.cc:1034] running on GCE node
I20260619 12:02:47.166337 16190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:47.166618 16190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:47.167779 16190 hybrid_clock.cc:648] HybridClock initialized: now 1781870567167761 us; error 43 us; skew 500 ppm
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:47.170513 16190 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:02:47.171008 16190 webserver.cc:460] Webserver started at http://127.0.203.254:42595/ using document root <none> and password file <none>
I20260619 12:02:47.171280 16190 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:47.171351 16190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:47.171504 16190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:47.172590 16190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "6b28349953974e949a83844850ccbe63"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9241b279fcdbc3a6e3fe44114d0d7e1a"
server_key_iv: "69f7a76a29cb48adb0d138b4da7903b5"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.173043 16190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "6b28349953974e949a83844850ccbe63"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9241b279fcdbc3a6e3fe44114d0d7e1a"
server_key_iv: "69f7a76a29cb48adb0d138b4da7903b5"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.174549 16190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:47.175637 16205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.175920 16190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:47.176075 16190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "6b28349953974e949a83844850ccbe63"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9241b279fcdbc3a6e3fe44114d0d7e1a"
server_key_iv: "69f7a76a29cb48adb0d138b4da7903b5"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.176165 16190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:47.185487 16190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:47.187798 16190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:47.187974 16190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:47.193008 16190 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:44921
I20260619 12:02:47.193066 16257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:44921 every 8 connection(s)
I20260619 12:02:47.193459 16190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:02:47.194388 16258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:47.197398 16258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Bootstrap starting.
I20260619 12:02:47.197515 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 16190
I20260619 12:02:47.197616 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:02:47.197860 815 external_mini_cluster.cc:1428] Setting key b86b9853d6f1e98cc9d46e3b67275430
I20260619 12:02:47.198269 16258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:47.198807 16258 log.cc:824] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:47.199640 16258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: No bootstrap required, opened a new log
I20260619 12:02:47.202165 16258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:02:47.202364 16258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:47.202414 16258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b28349953974e949a83844850ccbe63, State: Initialized, Role: FOLLOWER
I20260619 12:02:47.202601 16258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:02:47.202704 16258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:02:47.202780 16258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:02:47.202843 16258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:47.203665 16258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:02:47.203903 16258 leader_election.cc:302] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b28349953974e949a83844850ccbe63; no voters:
I20260619 12:02:47.204136 16258 leader_election.cc:288] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:02:47.204232 16263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:47.204504 16258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:02:47.204521 16263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 LEADER]: Becoming Leader. State: Replica: 6b28349953974e949a83844850ccbe63, State: Running, Role: LEADER
I20260619 12:02:47.204871 16263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:47.205760 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b28349953974e949a83844850ccbe63. Latest consensus state: current_term: 1 leader_uuid: "6b28349953974e949a83844850ccbe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } } }
I20260619 12:02:47.205850 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:47.206111 16264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b28349953974e949a83844850ccbe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } } }
I20260619 12:02:47.206259 16264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: This master's current role is: LEADER
I20260619 12:02:47.206300 16274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:02:47.206903 16274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:02:47.208823 16274 catalog_manager.cc:1347] Generated new cluster ID: 521480b04ad54670b02174d04c2a682a
I20260619 12:02:47.208875 16274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:02:47.209844 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.198441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.0.203.254:44921)
0619 12:02:47.198694 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.198707 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.198744 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:02:47.199514 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.199522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.199576 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.199732 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:47.200347 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.201332 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.202093 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.202623 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.202748 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.206058 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:47.206062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:47.206063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:47.206075 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:47.208055 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.208542 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.208545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.208546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.208685 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.208898 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.208899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.208899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.209147 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:47.209314 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.209533 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.209686 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260619 12:02:47.223429 16274 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:02:47.224148 16274 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:02:47.232110 16274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Generated new TSK 0
I20260619 12:02:47.232307 16274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:02:47.264460 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44921
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:47.349815 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:47.350018 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:47.350042 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:47.351192 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:47.351233 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:47.351259 16284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:02:47.353237 16284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:47.353487 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:47.353749 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:47.358843 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:47.358804 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:47.358964 16284 server_base.cc:1034] running on GCE node
W20260619 12:02:47.358884 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:47.359256 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:47.359548 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:47.360741 16284 hybrid_clock.cc:648] HybridClock initialized: now 1781870567360733 us; error 34 us; skew 500 ppm
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:47.363471 16284 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:02:47.363941 16284 webserver.cc:460] Webserver started at http://127.0.203.193:35219/ using document root <none> and password file <none>
I20260619 12:02:47.364207 16284 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:47.364280 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:47.364428 16284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:47.365559 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "20b85c7fe8004f5cbd879c10569a7676"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "52d97ce240cdd7db00884c40aedeb674"
server_key_iv: "c2a40a2d5d048c38595037c68c1d6722"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.365921 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "20b85c7fe8004f5cbd879c10569a7676"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "52d97ce240cdd7db00884c40aedeb674"
server_key_iv: "c2a40a2d5d048c38595037c68c1d6722"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.367282 16284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:02:47.368084 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.368300 16284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:47.368399 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "20b85c7fe8004f5cbd879c10569a7676"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "52d97ce240cdd7db00884c40aedeb674"
server_key_iv: "c2a40a2d5d048c38595037c68c1d6722"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.368482 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:47.399866 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:47.402390 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:47.402566 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:47.402823 16284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:47.403149 16284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:47.403203 16284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.403244 16284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:47.403272 16284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.409379 16284 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:43585
I20260619 12:02:47.409509 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:43585 every 8 connection(s)
I20260619 12:02:47.409780 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:02:47.409994 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 16284
I20260619 12:02:47.410100 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:02:47.410358 815 external_mini_cluster.cc:1428] Setting key 78f356c86ae7fdf12aa2666a84f49c5e
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:47.420120 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.410553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:59627 (local address 127.0.203.254:44921)
0619 12:02:47.410621 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.410624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.410851 (+ 227us) server_negotiation.cc:407] Connection header received
0619 12:02:47.411493 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.411494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.411503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.411570 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:47.412926 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.413434 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.414423 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.414740 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.414812 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.417071 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:47.417073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:47.417073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:47.417088 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:47.418693 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.419111 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.419113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.419113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.419172 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.419439 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.419440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.419440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.419592 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:47.419674 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.419926 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.420009 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 12:02:47.420168 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.410542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:59627)
0619 12:02:47.410726 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.410771 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.411393 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.411649 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.411653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.412476 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:47.412809 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.412816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.413531 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.413532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.414334 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.414336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.414846 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.414847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.414960 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.415561 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:47.415569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:47.416974 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:47.418785 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:47.418792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:47.418794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:47.419024 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:47.419266 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:47.419267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:47.419267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:47.419367 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:47.419664 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:47.419668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:47.419792 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.419876 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.420001 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
I20260619 12:02:47.421036 16413 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44921
I20260619 12:02:47.421123 16413 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:47.421352 16413 heartbeater.cc:511] Master 127.0.203.254:44921 requested a full tablet report, sending...
I20260619 12:02:47.421892 16222 ts_manager.cc:194] Registered new tserver with Master: 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193:43585)
I20260619 12:02:47.422513 16222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:59627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:02:47.440085 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44921
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:47.521024 16418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:47.521234 16418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:47.521260 16418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:47.522339 16418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:47.522379 16418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:47.522405 16418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:02:47.523821 16418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:47.524097 16418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:47.524307 16418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:47.529541 16423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:47.529541 16424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:47.529536 16426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:47.529573 16418 server_base.cc:1034] running on GCE node
I20260619 12:02:47.529975 16418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:47.530292 16418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:47.531467 16418 hybrid_clock.cc:648] HybridClock initialized: now 1781870567531462 us; error 37 us; skew 500 ppm
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:47.534222 16418 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:02:47.534744 16418 webserver.cc:460] Webserver started at http://127.0.203.194:36835/ using document root <none> and password file <none>
I20260619 12:02:47.534983 16418 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:47.535104 16418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:47.535424 16418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:47.536561 16418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "3bd1829110584f3cba46edc4ba528933"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "56b0e4c8a33b7ab4f74f3e51fac8664b"
server_key_iv: "effe01883d4378a9a66100926593fe38"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.536998 16418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "3bd1829110584f3cba46edc4ba528933"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "56b0e4c8a33b7ab4f74f3e51fac8664b"
server_key_iv: "effe01883d4378a9a66100926593fe38"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.538372 16418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:47.539222 16433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.539462 16418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:47.539570 16418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "3bd1829110584f3cba46edc4ba528933"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "56b0e4c8a33b7ab4f74f3e51fac8664b"
server_key_iv: "effe01883d4378a9a66100926593fe38"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.539669 16418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:47.549937 16418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:47.551709 16418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:47.551863 16418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:47.552083 16418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:47.552392 16418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:47.552448 16418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.552489 16418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:47.552517 16418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.558610 16418 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:37103
I20260619 12:02:47.558647 16546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:37103 every 8 connection(s)
I20260619 12:02:47.558967 16418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:47.565258 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 16418
I20260619 12:02:47.565371 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:02:47.565548 815 external_mini_cluster.cc:1428] Setting key 7c9acee28911509edd65147bd0e24c61
I20260619 12:02:47.568837 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.559707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:41879 (local address 127.0.203.254:44921)
0619 12:02:47.559778 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.559780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.560004 (+ 224us) server_negotiation.cc:407] Connection header received
0619 12:02:47.560470 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.560471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.560480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.560532 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:47.561516 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.562014 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.562988 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.563218 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.563278 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.565695 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:47.565697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:47.565697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:47.565706 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:47.567105 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.567641 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.567642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.567643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.567701 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.568029 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.568030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.568030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.568217 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:47.568293 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.568661 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.568745 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 12:02:47.569032 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.559699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:41879)
0619 12:02:47.559877 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.559913 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.560386 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.560583 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.560586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.561237 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:47.561434 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.561439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.562094 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.562094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.562899 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.562901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.563316 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.563317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.563421 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.563979 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:47.563987 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:47.565570 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:47.567238 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:47.567245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:47.567247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:47.567557 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:47.567770 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:47.567771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:47.567772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:47.567915 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:47.568299 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:47.568305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:47.568485 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.568594 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.568789 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260619 12:02:47.569522 16547 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44921
I20260619 12:02:47.569628 16547 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:47.569818 16547 heartbeater.cc:511] Master 127.0.203.254:44921 requested a full tablet report, sending...
I20260619 12:02:47.570312 16222 ts_manager.cc:194] Registered new tserver with Master: 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103)
I20260619 12:02:47.570922 16222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:41879
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:02:47.595238 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44921
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:02:47.677738 16552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:02:47.677999 16552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:02:47.678050 16552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:02:47.679543 16552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:02:47.679618 16552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:02:47.679665 16552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:02:47.681232 16552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:02:47.681500 16552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:02:47.681771 16552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:02:47.687100 16557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:02:47.687084 16560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:47.687100 16552 server_base.cc:1034] running on GCE node
W20260619 12:02:47.687084 16558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:02:47.687548 16552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:02:47.687824 16552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:02:47.688987 16552 hybrid_clock.cc:648] HybridClock initialized: now 1781870567688974 us; error 31 us; skew 500 ppm
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:02:47.691505 16552 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:02:47.691951 16552 webserver.cc:460] Webserver started at http://127.0.203.195:45049/ using document root <none> and password file <none>
I20260619 12:02:47.692138 16552 fs_manager.cc:359] Metadata directory not provided
I20260619 12:02:47.692179 16552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:02:47.692267 16552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:02:47.693354 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "b15af04dd98547a9b9daa7135d9842be"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9605ac9f2a54668b30c13bf457d3e3f7"
server_key_iv: "1c90ff772d9895a9325bb659201dd06f"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.693711 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "b15af04dd98547a9b9daa7135d9842be"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9605ac9f2a54668b30c13bf457d3e3f7"
server_key_iv: "1c90ff772d9895a9325bb659201dd06f"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.695113 16552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:02:47.695916 16567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.696172 16552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:02:47.696249 16552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "b15af04dd98547a9b9daa7135d9842be"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9605ac9f2a54668b30c13bf457d3e3f7"
server_key_iv: "1c90ff772d9895a9325bb659201dd06f"
server_key_version: "encryptionkey@0"
I20260619 12:02:47.696309 16552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:02:47.720443 16552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:02:47.722868 16552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:02:47.722985 16552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:02:47.723169 16552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:02:47.723500 16552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:02:47.723542 16552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.723588 16552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:02:47.723619 16552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:02:47.729614 16552 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:36203
I20260619 12:02:47.729688 16680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:36203 every 8 connection(s)
I20260619 12:02:47.730005 16552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:02:47.730706 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 16552
I20260619 12:02:47.730801 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:02:47.730973 815 external_mini_cluster.cc:1428] Setting key bc2f86b5007e4ca11aeb11de7df9c9dd
Jun 19 12:02:47 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:02:47.740463 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.731250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:34233 (local address 127.0.203.254:44921)
0619 12:02:47.731300 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.731302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.731578 (+ 276us) server_negotiation.cc:407] Connection header received
0619 12:02:47.732750 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.732751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.732763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.732825 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:47.733475 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.733988 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.734996 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.735207 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.735289 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.737595 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:47.737597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:47.737598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:47.737605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:47.738987 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.739446 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.739447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.739448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.739511 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.739762 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.739763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.739764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.739880 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:47.739957 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.740285 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.740363 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:02:47.740721 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.731189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:34233)
0619 12:02:47.731410 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.731472 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.732595 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.732895 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.732901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.733170 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:02:47.733377 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.733381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.734088 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.734089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.734904 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.734906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.735297 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.735298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.735423 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.735924 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:02:47.735932 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:02:47.737495 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:02:47.739127 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:47.739134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:47.739136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:47.739363 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:47.739584 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:02:47.739585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:02:47.739585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:02:47.739695 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:02:47.739949 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:02:47.739955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:02:47.740094 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.740261 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.740428 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260619 12:02:47.741228 16681 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44921
I20260619 12:02:47.741309 16681 heartbeater.cc:464] Registering TS with master...
I20260619 12:02:47.741447 16681 heartbeater.cc:511] Master 127.0.203.254:44921 requested a full tablet report, sending...
I20260619 12:02:47.741755 16222 ts_manager.cc:194] Registered new tserver with Master: b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
I20260619 12:02:47.742152 16222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:34233
I20260619 12:02:47.742872 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:02:47.750489 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.744305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.0.203.254:44921)
0619 12:02:47.744409 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.744411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.744513 (+ 102us) server_negotiation.cc:407] Connection header received
0619 12:02:47.744599 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.744599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.744609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.744689 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:02:47.745254 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.745728 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.746433 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.746638 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.746712 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.747960 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:02:47.747962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:02:47.747963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:02:47.747970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:02:47.749325 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.749691 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.749692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.749692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.749751 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:02:47.749951 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:02:47.749951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:02:47.749952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:02:47.750097 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:02:47.750208 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.750305 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.750382 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 12:02:47.752230 16222 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:02:47.753924 16222 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:02:47.760622 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.756713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.0.203.195:36203)
0619 12:02:47.757233 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.757238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.757246 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:02:47.757286 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.757288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.757304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.757379 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:47.757515 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.758516 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.759506 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.760239 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.760317 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.760373 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.760520 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.760552 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":106,"threads_started":1}
I20260619 12:02:47.760646 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.756296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37103 (local address 127.0.0.1:37640)
0619 12:02:47.756434 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.756471 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.756537 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.757004 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.757007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.757010 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:47.757132 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.757134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.758188 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.758189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.759695 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.759697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.760430 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.760432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.760493 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.760496 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.760539 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.760571 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
I20260619 12:02:47.760782 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.756494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36203 (local address 127.0.0.1:45618)
0619 12:02:47.756680 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.756689 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.756729 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.757391 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.757392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.757394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:47.757428 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.757429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.758613 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.758614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.759445 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.759448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.760337 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.760338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.760412 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.760426 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.760495 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.760561 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":57,"threads_started":1}
I20260619 12:02:47.760928 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.756296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:43585 (local address 127.0.0.1:52476)
0619 12:02:47.756434 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.756471 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.756536 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.757047 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.757048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.757051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:47.757125 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.757127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.758396 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.758397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.759491 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.759493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.760679 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.760680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.760752 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.760756 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.760811 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.760869 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":28,"thread_start_us":53,"threads_started":1}
I20260619 12:02:47.761050 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.756607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.0.203.194:37103)
0619 12:02:47.756829 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.756834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.756845 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:02:47.756886 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.756888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.756908 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.756973 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:47.757225 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.758085 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.759772 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.760252 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.760341 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.760399 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.760898 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.760947 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":166,"threads_started":1}
I20260619 12:02:47.761201 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.756409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.0.203.193:43585)
0619 12:02:47.756858 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.756863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.756872 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:02:47.756908 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.756910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.756925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.757029 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:47.757193 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.758294 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.759571 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.760239 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.760316 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.760373 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.761076 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.761126 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":78,"threads_started":1}
I20260619 12:02:47.761901 16347 tablet_service.cc:1467] Processing CreateTablet for tablet 01d91140b42d4f3ea5154d41a421c3ec (DEFAULT_TABLE table=test-table [id=7cd18bbd863646a2a312432bfe6b0e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:47.761906 16481 tablet_service.cc:1467] Processing CreateTablet for tablet 01d91140b42d4f3ea5154d41a421c3ec (DEFAULT_TABLE table=test-table [id=7cd18bbd863646a2a312432bfe6b0e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:47.761965 16615 tablet_service.cc:1467] Processing CreateTablet for tablet 01d91140b42d4f3ea5154d41a421c3ec (DEFAULT_TABLE table=test-table [id=7cd18bbd863646a2a312432bfe6b0e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:02:47.762253 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d91140b42d4f3ea5154d41a421c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:47.762254 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d91140b42d4f3ea5154d41a421c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:47.762255 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d91140b42d4f3ea5154d41a421c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:02:47.765445 16699 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Bootstrap starting.
I20260619 12:02:47.765743 16698 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap starting.
I20260619 12:02:47.766096 16699 tablet_bootstrap.cc:654] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:47.766316 16698 tablet_bootstrap.cc:654] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:47.766639 16699 log.cc:824] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:47.766816 16698 log.cc:824] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:47.766980 16700 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Bootstrap starting.
I20260619 12:02:47.767360 16699 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: No bootstrap required, opened a new log
I20260619 12:02:47.767416 16699 ts_tablet_manager.cc:1397] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:47.767532 16698 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: No bootstrap required, opened a new log
I20260619 12:02:47.767618 16698 ts_tablet_manager.cc:1397] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:02:47.769163 16699 raft_consensus.cc:348] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.769316 16699 raft_consensus.cc:374] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:47.769340 16699 raft_consensus.cc:729] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15af04dd98547a9b9daa7135d9842be, State: Initialized, Role: FOLLOWER
I20260619 12:02:47.769222 16698 raft_consensus.cc:348] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.769381 16698 raft_consensus.cc:374] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:47.769425 16698 raft_consensus.cc:729] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b85c7fe8004f5cbd879c10569a7676, State: Initialized, Role: FOLLOWER
I20260619 12:02:47.769487 16699 consensus_queue.cc:260] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.769536 16698 consensus_queue.cc:260] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.769812 16681 heartbeater.cc:503] Master 127.0.203.254:44921 was elected leader, sending a full tablet report...
I20260619 12:02:47.769847 16413 heartbeater.cc:503] Master 127.0.203.254:44921 was elected leader, sending a full tablet report...
I20260619 12:02:47.770077 16698 ts_tablet_manager.cc:1428] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 12:02:47.770085 16699 ts_tablet_manager.cc:1428] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:02:47.770851 16700 tablet_bootstrap.cc:654] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Neither blocks nor log segments found. Creating new log.
I20260619 12:02:47.771458 16700 log.cc:824] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Log is configured to *not* fsync() on all Append() calls
I20260619 12:02:47.772149 16700 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: No bootstrap required, opened a new log
I20260619 12:02:47.772231 16700 ts_tablet_manager.cc:1397] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:02:47.773669 16700 raft_consensus.cc:348] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.773837 16700 raft_consensus.cc:374] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:02:47.773883 16700 raft_consensus.cc:729] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd1829110584f3cba46edc4ba528933, State: Initialized, Role: FOLLOWER
I20260619 12:02:47.774003 16700 consensus_queue.cc:260] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.774260 16700 ts_tablet_manager.cc:1428] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:02:47.774314 16547 heartbeater.cc:503] Master 127.0.203.254:44921 was elected leader, sending a full tablet report...
W20260619 12:02:47.809588 16548 tablet.cc:2369] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:47.890288 16704 raft_consensus.cc:482] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:02:47.890436 16704 raft_consensus.cc:504] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.890858 16704 leader_election.cc:288] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103), b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
I20260619 12:02:47.894124 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.891300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:44107 (local address 127.0.203.194:37103)
0619 12:02:47.891357 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.891359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.891369 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:02:47.891403 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.891404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.891415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.891494 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:47.891680 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.892252 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.893308 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.893727 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.893785 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.893792 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.894007 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.894049 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:02:47.894124 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.891173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37103 (local address 127.0.203.193:44107)
0619 12:02:47.891233 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.891246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.891294 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.891507 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.891508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.891510 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:47.891592 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.891595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.892327 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.892328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.893221 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.893224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.893814 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.893816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.893931 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.893936 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.893997 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.894045 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 12:02:47.894338 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.891335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36203 (local address 127.0.203.193:38101)
0619 12:02:47.891454 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.891465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:02:47.891514 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:02:47.891680 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:02:47.891681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:02:47.891682 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:02:47.891754 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.891756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.892463 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.892464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.893290 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:02:47.893292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.894045 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.894046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:02:47.894135 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.894140 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:02:47.894218 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:02:47.894261 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260619 12:02:47.894351 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.891438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:38101 (local address 127.0.203.195:36203)
0619 12:02:47.891500 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.891502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.891510 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:02:47.891591 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.891592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.891603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.891655 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:02:47.891828 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.892377 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.893412 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.893864 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.893947 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.893956 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.894244 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.894289 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:02:47.894743 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d91140b42d4f3ea5154d41a421c3ec" candidate_uuid: "20b85c7fe8004f5cbd879c10569a7676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd1829110584f3cba46edc4ba528933" is_pre_election: true
I20260619 12:02:47.894933 16501 raft_consensus.cc:2454] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b85c7fe8004f5cbd879c10569a7676 in term 0.
I20260619 12:02:47.894896 16635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d91140b42d4f3ea5154d41a421c3ec" candidate_uuid: "20b85c7fe8004f5cbd879c10569a7676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15af04dd98547a9b9daa7135d9842be" is_pre_election: true
I20260619 12:02:47.895045 16635 raft_consensus.cc:2454] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b85c7fe8004f5cbd879c10569a7676 in term 0.
I20260619 12:02:47.895167 16301 leader_election.cc:302] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676, 3bd1829110584f3cba46edc4ba528933; no voters:
I20260619 12:02:47.895323 16704 raft_consensus.cc:2781] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:02:47.895383 16704 raft_consensus.cc:482] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:02:47.895409 16704 raft_consensus.cc:3037] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:47.896114 16704 raft_consensus.cc:504] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.896276 16704 leader_election.cc:288] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 1 election: Requested vote from peers 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103), b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
I20260619 12:02:47.896440 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d91140b42d4f3ea5154d41a421c3ec" candidate_uuid: "20b85c7fe8004f5cbd879c10569a7676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd1829110584f3cba46edc4ba528933"
I20260619 12:02:47.896472 16635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d91140b42d4f3ea5154d41a421c3ec" candidate_uuid: "20b85c7fe8004f5cbd879c10569a7676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15af04dd98547a9b9daa7135d9842be"
I20260619 12:02:47.896543 16501 raft_consensus.cc:3037] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:47.896545 16635 raft_consensus.cc:3037] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:02:47.897287 16635 raft_consensus.cc:2454] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b85c7fe8004f5cbd879c10569a7676 in term 1.
I20260619 12:02:47.897315 16501 raft_consensus.cc:2454] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b85c7fe8004f5cbd879c10569a7676 in term 1.
I20260619 12:02:47.897486 16301 leader_election.cc:302] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676, b15af04dd98547a9b9daa7135d9842be; no voters:
I20260619 12:02:47.897599 16704 raft_consensus.cc:2781] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:02:47.897764 16704 raft_consensus.cc:686] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 1 LEADER]: Becoming Leader. State: Replica: 20b85c7fe8004f5cbd879c10569a7676, State: Running, Role: LEADER
I20260619 12:02:47.897866 16704 consensus_queue.cc:237] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:02:47.898804 16222 catalog_manager.cc:5495] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "20b85c7fe8004f5cbd879c10569a7676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } health_report { overall_health: UNKNOWN } } }
I20260619 12:02:47.909880 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.907453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.0.203.193:43585)
0619 12:02:47.907499 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:02:47.907503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:02:47.907515 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:02:47.907551 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:02:47.907552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:02:47.907569 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:02:47.907652 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:02:47.907746 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.908226 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.909081 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:02:47.909284 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:02:47.909361 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:02:47.909565 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:02:47.909672 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:02:47.909746 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:02:47.909779 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 12:02:47.909799 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 12:02:47.911181 16414 tablet.cc:2369] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:02:47.911830 16501 raft_consensus.cc:1261] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 1 FOLLOWER]: Refusing update from remote peer 20b85c7fe8004f5cbd879c10569a7676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:47.911870 16635 raft_consensus.cc:1261] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Refusing update from remote peer 20b85c7fe8004f5cbd879c10569a7676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:02:47.912223 16704 consensus_queue.cc:1035] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:47.912392 16708 consensus_queue.cc:1035] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:02:47.915210 16712 mvcc.cc:205] Tried to move back new op lower bound from 7298541846164144128 to 7298541846110449664. Current Snapshot: MvccSnapshot[applied={T|T < 7298541846164144128}]
I20260619 12:02:47.916999 16222 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:47.917112 16222 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:02:47.917904 16222 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Sending DeleteTablet for 3 replicas of tablet 01d91140b42d4f3ea5154d41a421c3ec
I20260619 12:02:47.918221 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d91140b42d4f3ea5154d41a421c3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:47 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:52476
I20260619 12:02:47.918329 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 16284
I20260619 12:02:47.918399 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d91140b42d4f3ea5154d41a421c3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:47 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:37640
I20260619 12:02:47.918598 16615 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d91140b42d4f3ea5154d41a421c3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:47 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:45618
I20260619 12:02:47.920084 16721 tablet_replica.cc:330] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: stopping tablet replica
I20260619 12:02:47.920192 16721 raft_consensus.cc:2229] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:47.920295 16721 raft_consensus.cc:2258] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:47.920439 16722 tablet_replica.cc:330] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: stopping tablet replica
I20260619 12:02:47.920542 16722 raft_consensus.cc:2229] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:02:47.920678 16722 raft_consensus.cc:2258] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:02:47.920804 16721 ts_tablet_manager.cc:1905] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:47.921088 16722 ts_tablet_manager.cc:1905] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:02:47.922493 16721 ts_tablet_manager.cc:1918] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:47.922592 16721 log.cc:1196] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/01d91140b42d4f3ea5154d41a421c3ec
I20260619 12:02:47.922736 16722 ts_tablet_manager.cc:1918] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:02:47.922809 16722 log.cc:1196] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/01d91140b42d4f3ea5154d41a421c3ec
I20260619 12:02:47.922942 16721 ts_tablet_manager.cc:1939] T 01d91140b42d4f3ea5154d41a421c3ec P 3bd1829110584f3cba46edc4ba528933: Deleting consensus metadata
I20260619 12:02:47.923136 16722 ts_tablet_manager.cc:1939] T 01d91140b42d4f3ea5154d41a421c3ec P b15af04dd98547a9b9daa7135d9842be: Deleting consensus metadata
I20260619 12:02:47.923424 16207 catalog_manager.cc:4841] TS 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103): tablet 01d91140b42d4f3ea5154d41a421c3ec (table test-table [id=7cd18bbd863646a2a312432bfe6b0e2d]) successfully deleted
I20260619 12:02:47.923532 16207 catalog_manager.cc:4841] TS b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203): tablet 01d91140b42d4f3ea5154d41a421c3ec (table test-table [id=7cd18bbd863646a2a312432bfe6b0e2d]) successfully deleted
W20260619 12:02:47.924276 16208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:43585 (error 108)
I20260619 12:02:47.924480 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 16418
I20260619 12:02:47.925212 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:02:47.925041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:43585 (local address 127.0.0.1:52482)
0619 12:02:47.925141 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 12:02:47.925184 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 12:02:47.925396 16208 catalog_manager.cc:4584] TS 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193:43585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01d91140b42d4f3ea5154d41a421c3ec: Network error: Client connection negotiation failed: client connection to 127.0.203.193:43585: connect: Connection refused (error 111)
I20260619 12:02:47.929687 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 16552
I20260619 12:02:47.934477 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 16190
I20260619 12:03:07.940086 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44921
--webserver_interface=127.0.203.254
--webserver_port=42595
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:08.028107 16724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:08.028343 16724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:08.028365 16724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:08.029644 16724 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:03:08.029706 16724 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:03:08.029719 16724 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:03:08.029731 16724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:08.029744 16724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:08.029758 16724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:08.029771 16724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:08.029811 16724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:08.031347 16724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=42595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:08.031621 16724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:08.031898 16724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:08.042739 16733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:08.042765 16730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:08.042938 16724 server_base.cc:1034] running on GCE node
W20260619 12:03:08.042795 16731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:08.043421 16724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:08.043926 16724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:08.045171 16724 hybrid_clock.cc:648] HybridClock initialized: now 1781870588045139 us; error 46 us; skew 500 ppm
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:08.054498 16724 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:03:08.055262 16724 webserver.cc:460] Webserver started at http://127.0.203.254:42595/ using document root <none> and password file <none>
I20260619 12:03:08.055619 16724 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:08.055696 16724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:08.057617 16724 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:03:08.058645 16740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.058923 16724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:08.059053 16724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "6b28349953974e949a83844850ccbe63"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9241b279fcdbc3a6e3fe44114d0d7e1a"
server_key_iv: "69f7a76a29cb48adb0d138b4da7903b5"
server_key_version: "encryptionkey@0"
I20260619 12:03:08.059548 16724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:08.069851 16724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:08.072234 16724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:08.072361 16724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:08.077193 16724 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:44921
I20260619 12:03:08.077289 16792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:44921 every 8 connection(s)
I20260619 12:03:08.077730 16724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:08.079631 16793 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:03:08.080348 16793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Bootstrap starting.
I20260619 12:03:08.086725 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 16724
I20260619 12:03:08.087291 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:43585
--local_ip_for_outbound_sockets=127.0.203.193
--tserver_master_addrs=127.0.203.254:44921
--webserver_port=35219
--webserver_interface=127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:03:08.091483 16793 log.cc:824] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:08.093925 16793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:03:08.094257 16793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Bootstrap complete.
I20260619 12:03:08.097052 16793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:08.097337 16793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b28349953974e949a83844850ccbe63, State: Initialized, Role: FOLLOWER
I20260619 12:03:08.097553 16793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:08.097640 16793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:08.097702 16793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:08.097755 16793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:03:08.098953 16793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:08.099092 16793 leader_election.cc:302] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b28349953974e949a83844850ccbe63; no voters:
I20260619 12:03:08.099337 16793 leader_election.cc:288] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:03:08.099382 16797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:03:08.099527 16797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 LEADER]: Becoming Leader. State: Replica: 6b28349953974e949a83844850ccbe63, State: Running, Role: LEADER
I20260619 12:03:08.099677 16793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:08.099712 16797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:08.100234 16799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b28349953974e949a83844850ccbe63. Latest consensus state: current_term: 2 leader_uuid: "6b28349953974e949a83844850ccbe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } } }
I20260619 12:03:08.100256 16798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b28349953974e949a83844850ccbe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } } }
I20260619 12:03:08.100477 16799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:08.100522 16798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:08.101044 16806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:08.102072 16806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:08.102186 16806 catalog_manager.cc:1259] Loaded cluster ID: 521480b04ad54670b02174d04c2a682a
I20260619 12:03:08.102257 16806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:08.103314 16806 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:08.103533 16806 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Loaded TSK: 0
I20260619 12:03:08.127947 16806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Generated new TSK 1
I20260619 12:03:08.128114 16806 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:03:08.173799 16795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:08.174064 16795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:08.174106 16795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:08.175447 16795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:08.175521 16795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:08.175570 16795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:08.177170 16795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:43585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=35219
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:08.177469 16795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:08.177722 16795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:08.184342 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:08.184346 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:08.184496 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:08.184844 16795 server_base.cc:1034] running on GCE node
I20260619 12:03:08.185105 16795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:08.185410 16795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:08.186606 16795 hybrid_clock.cc:648] HybridClock initialized: now 1781870588186588 us; error 28 us; skew 500 ppm
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:08.190340 16795 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:03:08.190899 16795 webserver.cc:460] Webserver started at http://127.0.203.193:35219/ using document root <none> and password file <none>
I20260619 12:03:08.191162 16795 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:08.191254 16795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:08.192909 16795 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:08.193815 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.194078 16795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:08.194198 16795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "20b85c7fe8004f5cbd879c10569a7676"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "52d97ce240cdd7db00884c40aedeb674"
server_key_iv: "c2a40a2d5d048c38595037c68c1d6722"
server_key_version: "encryptionkey@0"
I20260619 12:03:08.194584 16795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:08.240900 16795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:08.243427 16795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:08.243592 16795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:08.243848 16795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:08.244395 16837 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:03:08.245940 16795 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:03:08.246004 16795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 12:03:08.246032 16795 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:03:08.246699 16795 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:03:08.246750 16795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 12:03:08.246784 16837 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap starting.
I20260619 12:03:08.255987 16795 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:43585
I20260619 12:03:08.256112 16944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:43585 every 8 connection(s)
I20260619 12:03:08.256563 16795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:03:08.258466 16837 log.cc:824] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:08.261294 16837 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:03:08.261611 16837 tablet_bootstrap.cc:492] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap complete.
I20260619 12:03:08.262861 16837 ts_tablet_manager.cc:1397] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260619 12:03:08.263389 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 16795
I20260619 12:03:08.263681 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:37103
--local_ip_for_outbound_sockets=127.0.203.194
--tserver_master_addrs=127.0.203.254:44921
--webserver_port=36835
--webserver_interface=127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:03:08.264513 16837 raft_consensus.cc:348] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:03:08.264957 16837 raft_consensus.cc:729] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b85c7fe8004f5cbd879c10569a7676, State: Initialized, Role: FOLLOWER
I20260619 12:03:08.265117 16837 consensus_queue.cc:260] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } } peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } }
I20260619 12:03:08.265458 16837 ts_tablet_manager.cc:1428] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:08.275764 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.257538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:55961)
0619 12:03:08.257797 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.257860 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.258451 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.259286 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.259292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.259652 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:08.259936 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.259943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.261284 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.261285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.262747 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.262750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.263633 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.263635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.263985 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.268979 (+ 4994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:08.268994 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:08.270967 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:08.273588 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.273598 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.273600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.273892 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.274251 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.274252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.274252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.274368 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.274947 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:08.274953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:08.275106 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.275214 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.275498 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":133,"threads_started":1}
I20260619 12:03:08.275791 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.257623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:55961 (local address 127.0.203.254:44921)
0619 12:03:08.258675 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.258707 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.258746 (+ 39us) server_negotiation.cc:407] Connection header received
0619 12:03:08.258896 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.258902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.258958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.259114 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:08.260108 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.261150 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.262856 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.263436 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.263641 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.271110 (+ 7469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:08.271117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:08.271117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:08.271133 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:08.273467 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.274000 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.274004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.274004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.274144 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.274460 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.274462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.274462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.274853 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:08.274960 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.275317 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.275505 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":141,"threads_started":1}
I20260619 12:03:08.276814 16945 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44921
I20260619 12:03:08.276942 16945 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:08.277168 16945 heartbeater.cc:511] Master 127.0.203.254:44921 requested a full tablet report, sending...
I20260619 12:03:08.278136 16757 ts_manager.cc:194] Registered new tserver with Master: 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193:43585)
I20260619 12:03:08.279410 16757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:55961
I20260619 12:03:08.282243 16945 heartbeater.cc:503] Master 127.0.203.254:44921 was elected leader, sending a full tablet report...
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.193@KRBTEST.COM
I20260619 12:03:08.290652 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.278866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:43585 (local address 127.0.0.1:51742)
0619 12:03:08.279279 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.279322 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.279442 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.279828 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.279831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.280374 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:08.280451 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.280455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.281921 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.281922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.282840 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.282843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.283549 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.283551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.283705 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.284532 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:08.284539 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:08.286575 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:08.289179 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.289187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.289190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.289479 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.289794 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.289795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.289795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.289860 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.290202 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:08.290208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:08.290269 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.290375 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.290538 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":128,"threads_started":1}
I20260619 12:03:08.290652 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.278970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.0.203.193:43585)
0619 12:03:08.279192 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.279197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.279397 (+ 200us) server_negotiation.cc:407] Connection header received
0619 12:03:08.279607 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.279610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.279637 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.279681 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:08.280565 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.281805 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.282941 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.283436 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.283517 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.286711 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:08.286723 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:08.286723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:08.286733 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:08.289053 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.289597 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.289601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.289602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.289687 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.289931 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.289932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.289933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.290119 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:08.290220 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.290421 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.290539 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":118,"threads_started":1}
I20260619 12:03:08.291108 16879 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d91140b42d4f3ea5154d41a421c3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:02:47 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:51742
I20260619 12:03:08.291679 16954 tablet_replica.cc:330] stopping tablet replica
I20260619 12:03:08.291811 16954 raft_consensus.cc:2229] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:08.291893 16954 raft_consensus.cc:2258] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:08.292259 16954 ts_tablet_manager.cc:1905] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:08.294682 16954 ts_tablet_manager.cc:1918] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:08.294749 16954 log.cc:1196] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal/wals/01d91140b42d4f3ea5154d41a421c3ec
I20260619 12:03:08.295017 16954 ts_tablet_manager.cc:1939] T 01d91140b42d4f3ea5154d41a421c3ec P 20b85c7fe8004f5cbd879c10569a7676: Deleting consensus metadata
I20260619 12:03:08.295603 16743 catalog_manager.cc:4841] TS 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193:43585): tablet 01d91140b42d4f3ea5154d41a421c3ec (table test-table [id=7cd18bbd863646a2a312432bfe6b0e2d]) successfully deleted
W20260619 12:03:08.357810 16950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:08.358072 16950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:08.358112 16950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:08.359349 16950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:08.359412 16950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:08.359455 16950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:08.360937 16950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:37103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=36835
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:08.361208 16950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:08.361460 16950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:08.366860 16962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:08.366940 16950 server_base.cc:1034] running on GCE node
W20260619 12:03:08.366850 16960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:08.366833 16959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:08.367398 16950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:08.367658 16950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:08.368808 16950 hybrid_clock.cc:648] HybridClock initialized: now 1781870588368793 us; error 39 us; skew 500 ppm
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:08.371620 16950 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:03:08.372123 16950 webserver.cc:460] Webserver started at http://127.0.203.194:36835/ using document root <none> and password file <none>
I20260619 12:03:08.372390 16950 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:08.372483 16950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:08.374287 16950 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:08.375061 16969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.375340 16950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:08.375476 16950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "3bd1829110584f3cba46edc4ba528933"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "56b0e4c8a33b7ab4f74f3e51fac8664b"
server_key_iv: "effe01883d4378a9a66100926593fe38"
server_key_version: "encryptionkey@0"
I20260619 12:03:08.376060 16950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:08.402639 16950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:08.405162 16950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:08.405345 16950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:08.405650 16950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:08.406026 16950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:08.406086 16950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.406127 16950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:08.406176 16950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.413906 16950 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:37103
I20260619 12:03:08.413959 17082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:37103 every 8 connection(s)
I20260619 12:03:08.414355 16950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:03:08.420300 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 16950
I20260619 12:03:08.420703 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:36203
--local_ip_for_outbound_sockets=127.0.203.195
--tserver_master_addrs=127.0.203.254:44921
--webserver_port=45049
--webserver_interface=127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:08.426792 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.415168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:53425 (local address 127.0.203.254:44921)
0619 12:03:08.415222 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.415225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.415514 (+ 289us) server_negotiation.cc:407] Connection header received
0619 12:03:08.416152 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.416153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.416172 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.416228 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:08.417377 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.417937 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.419015 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.419242 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.419321 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.422576 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:08.422578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:08.422579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:08.422588 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:08.424549 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.425219 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.425222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.425223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.425317 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.425710 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.425711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.425711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.425858 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:08.425984 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.426560 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.426684 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 12:03:08.426999 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.415154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:53425)
0619 12:03:08.415349 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.415391 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.416031 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.416309 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.416314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.417045 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:08.417273 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.417278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.418044 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.418045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.418903 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.418905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.419325 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.419327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.419474 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.420193 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:08.420205 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:08.422417 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:08.424713 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.424725 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.424729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.425111 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.425413 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.425414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.425414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.425595 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.425933 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:08.426172 (+ 239us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:08.426318 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.426521 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.426719 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260619 12:03:08.427682 17083 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44921
I20260619 12:03:08.427801 17083 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:08.428071 17083 heartbeater.cc:511] Master 127.0.203.254:44921 requested a full tablet report, sending...
I20260619 12:03:08.429073 16757 ts_manager.cc:194] Registered new tserver with Master: 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103)
I20260619 12:03:08.429701 16757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:53425
W20260619 12:03:08.520344 17086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:08.520568 17086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:08.520596 17086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:08.521848 17086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:08.521909 17086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:08.521938 17086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:08.523442 17086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:36203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=45049
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:08.523660 17086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:08.523849 17086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:08.530283 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:08.530470 17086 server_base.cc:1034] running on GCE node
W20260619 12:03:08.530359 17092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:08.530264 17094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:08.531080 17086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:08.531440 17086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:08.532824 17086 hybrid_clock.cc:648] HybridClock initialized: now 1781870588532775 us; error 68 us; skew 500 ppm
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:08.536855 17086 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:03:08.537464 17086 webserver.cc:460] Webserver started at http://127.0.203.195:45049/ using document root <none> and password file <none>
I20260619 12:03:08.537956 17086 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:08.538143 17086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:08.539960 17086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:08.540959 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.541301 17086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:08.541436 17086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "b15af04dd98547a9b9daa7135d9842be"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9605ac9f2a54668b30c13bf457d3e3f7"
server_key_iv: "1c90ff772d9895a9325bb659201dd06f"
server_key_version: "encryptionkey@0"
I20260619 12:03:08.541945 17086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:08.553319 17086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:08.556442 17086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:08.556790 17086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:08.557092 17086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:08.557480 17086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:08.557541 17086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.557586 17086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:08.557611 17086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:08.565907 17086 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:36203
I20260619 12:03:08.565984 17214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:36203 every 8 connection(s)
I20260619 12:03:08.566468 17086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:03:08.568595 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 17086
Jun 19 12:03:08 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:08.578109 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.569335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.203.254:44921)
0619 12:03:08.569496 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.569498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.569630 (+ 132us) server_negotiation.cc:407] Connection header received
0619 12:03:08.569692 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.569693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.569703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.569841 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:08.570448 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.570942 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.571964 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.572279 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.572361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.574363 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:08.574365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:08.574365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:08.574375 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:08.576310 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.576959 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.576960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.576961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.577060 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.577438 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.577439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.577439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.577576 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:08.577729 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.577915 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.578027 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260619 12:03:08.579962 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.567402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:59583 (local address 127.0.203.254:44921)
0619 12:03:08.567522 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.567524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.567727 (+ 203us) server_negotiation.cc:407] Connection header received
0619 12:03:08.568275 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.568277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.568291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.568383 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:08.569772 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.570419 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.571813 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.572129 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.572224 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.575859 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:08.575861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:08.575861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:08.575869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:08.578107 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.578757 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.578759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.578759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.578833 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.579140 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.579141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.579142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.579269 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:08.579353 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.579747 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.579848 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:03:08.580215 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.567351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:59583)
0619 12:03:08.567534 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.567602 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.568128 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.568526 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.568531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.569341 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:08.569660 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.569668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.570525 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.570526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.571688 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.571691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.572260 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.572262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.572471 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.573310 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:08.573321 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:08.575734 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:08.578228 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.578239 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.578242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.578645 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.578914 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:08.578915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:08.578916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:08.579056 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:08.579360 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:08.579367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:08.579559 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.579711 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.579922 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260619 12:03:08.580914 17215 heartbeater.cc:347] Connected to a master server at 127.0.203.254:44921
I20260619 12:03:08.581043 17215 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:08.581295 17215 heartbeater.cc:511] Master 127.0.203.254:44921 requested a full tablet report, sending...
I20260619 12:03:08.581805 16757 ts_manager.cc:194] Registered new tserver with Master: b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
I20260619 12:03:08.582234 16757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:59583
I20260619 12:03:08.590451 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:03:08.593967 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.590927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.0.203.254:44921)
0619 12:03:08.591003 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.591005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.591057 (+ 52us) server_negotiation.cc:407] Connection header received
0619 12:03:08.591145 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.591145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.591201 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.591278 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:08.591536 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.592097 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.593098 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.593365 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.593441 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.593667 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:08.593726 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 12:03:08.593897 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51278: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 12:03:08.594066 16948 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51278: token expired
W20260619 12:03:08.594209 16684 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.203.254:44921)
I20260619 12:03:08.602986 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.594611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.0.203.254:44921)
0619 12:03:08.594682 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.594684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.594691 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:03:08.594710 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.594711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.594719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.594777 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:08.595303 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.595796 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.596555 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.596914 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.596986 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.599014 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:08.599015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:08.599016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:08.599024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:08.601631 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.602116 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.602117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.602118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.602184 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:08.602398 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:08.602399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:08.602399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:08.602555 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:08.602688 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.602772 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.602862 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:03:08.604854 16757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:03:08.607066 16757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:03:08.610273 16879 tablet_service.cc:1467] Processing CreateTablet for tablet d9f8c5705ad94ca09a37605f2bbcfe30 (DEFAULT_TABLE table=test-table [id=e8768fb73c3e443d90fa86244445b167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:08.610577 16879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f8c5705ad94ca09a37605f2bbcfe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:08.613816 16837 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap starting.
I20260619 12:03:08.614213 16837 tablet_bootstrap.cc:654] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:08.614794 16837 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: No bootstrap required, opened a new log
I20260619 12:03:08.614848 16837 ts_tablet_manager.cc:1397] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:08.615221 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.609806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37103 (local address 127.0.0.1:48784)
0619 12:03:08.610018 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.610030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.610075 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.610552 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.610553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.610556 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:08.610625 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.610627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.612468 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.612469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.613857 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.613860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.614875 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.614876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.614954 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.614980 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.615014 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.615073 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":66}
I20260619 12:03:08.615247 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.610336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.0.203.195:36203)
0619 12:03:08.610597 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.610604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.610813 (+ 209us) server_negotiation.cc:407] Connection header received
0619 12:03:08.610867 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.610869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.610888 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.611103 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:08.611417 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.612521 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.613962 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.614746 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.614824 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.614891 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.615122 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.615164 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":140,"threads_started":1}
I20260619 12:03:08.615480 16837 raft_consensus.cc:348] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.615630 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.609964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.203.194:37103)
0619 12:03:08.610269 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.610276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.610304 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:03:08.610363 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.610366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.610389 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.610507 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:08.610768 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.612351 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.614071 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.614783 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.615407 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.615469 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.615490 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.615527 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":163,"threads_started":1}
I20260619 12:03:08.615615 16837 raft_consensus.cc:374] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:08.615675 16837 raft_consensus.cc:729] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b85c7fe8004f5cbd879c10569a7676, State: Initialized, Role: FOLLOWER
I20260619 12:03:08.615821 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.609899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36203 (local address 127.0.0.1:58084)
0619 12:03:08.610667 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.610682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.610753 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.611140 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.611141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.611145 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:08.611193 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.611194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.612618 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.612619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.613857 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.613860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.614922 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.614923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.614969 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.614980 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.615029 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.615073 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":12,"thread_start_us":75,"threads_started":1}
I20260619 12:03:08.615823 16837 consensus_queue.cc:260] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.616075 16837 ts_tablet_manager.cc:1428] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:08.616364 17017 tablet_service.cc:1467] Processing CreateTablet for tablet d9f8c5705ad94ca09a37605f2bbcfe30 (DEFAULT_TABLE table=test-table [id=e8768fb73c3e443d90fa86244445b167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:08.616783 17017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f8c5705ad94ca09a37605f2bbcfe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:08.617467 17149 tablet_service.cc:1467] Processing CreateTablet for tablet d9f8c5705ad94ca09a37605f2bbcfe30 (DEFAULT_TABLE table=test-table [id=e8768fb73c3e443d90fa86244445b167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:08.617884 17149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f8c5705ad94ca09a37605f2bbcfe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:08.620175 17225 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Bootstrap starting.
I20260619 12:03:08.620975 17225 tablet_bootstrap.cc:654] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:08.621398 17226 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Bootstrap starting.
I20260619 12:03:08.621610 17225 log.cc:824] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:08.622219 17226 tablet_bootstrap.cc:654] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:08.622409 17225 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: No bootstrap required, opened a new log
I20260619 12:03:08.622465 17225 ts_tablet_manager.cc:1397] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:03:08.622923 17226 log.cc:824] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:08.623716 17226 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: No bootstrap required, opened a new log
I20260619 12:03:08.623809 17226 ts_tablet_manager.cc:1397] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:03:08.623796 17225 raft_consensus.cc:348] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.624001 17225 raft_consensus.cc:374] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:08.624058 17225 raft_consensus.cc:729] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd1829110584f3cba46edc4ba528933, State: Initialized, Role: FOLLOWER
I20260619 12:03:08.624192 17225 consensus_queue.cc:260] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.624507 17225 ts_tablet_manager.cc:1428] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:03:08.624590 17083 heartbeater.cc:503] Master 127.0.203.254:44921 was elected leader, sending a full tablet report...
I20260619 12:03:08.626222 17226 raft_consensus.cc:348] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.626428 17226 raft_consensus.cc:374] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:08.626478 17226 raft_consensus.cc:729] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15af04dd98547a9b9daa7135d9842be, State: Initialized, Role: FOLLOWER
I20260619 12:03:08.626622 17226 consensus_queue.cc:260] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.626955 17226 ts_tablet_manager.cc:1428] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:08.627028 17215 heartbeater.cc:503] Master 127.0.203.254:44921 was elected leader, sending a full tablet report...
W20260619 12:03:08.665143 17084 tablet.cc:2369] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:03:08.757539 16946 tablet.cc:2369] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:03:08.812326 17229 raft_consensus.cc:482] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:08.812502 17229 raft_consensus.cc:504] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.813058 17229 leader_election.cc:288] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203), 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193:43585)
I20260619 12:03:08.816838 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.813241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36203 (local address 127.0.203.194:51877)
0619 12:03:08.813307 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.813320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.813380 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.813584 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.813585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.813587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:08.813671 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.813673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.814651 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.814652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.815665 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.815667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.816453 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.816454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.816542 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.816546 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.816674 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.816737 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 12:03:08.817071 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.813353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:51877 (local address 127.0.203.195:36203)
0619 12:03:08.813430 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.813432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.813441 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:03:08.813471 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.813472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.813481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.813560 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:08.813797 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.814508 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.815807 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.816351 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.816414 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.816422 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.816859 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.816943 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:03:08.817291 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.813389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:43585 (local address 127.0.203.194:36809)
0619 12:03:08.813627 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.813647 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:08.813709 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:08.814009 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:08.814010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:08.814013 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:08.814109 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.814111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.815095 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.815095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.816045 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:08.816048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.817056 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.817057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:08.817125 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.817128 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:08.817172 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 12:03:08.817213 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
I20260619 12:03:08.817302 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.813475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:36809 (local address 127.0.203.193:43585)
0619 12:03:08.813699 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.813702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.813714 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:03:08.813862 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.813867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.813893 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.813975 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:08.814232 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.814943 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.816204 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.816974 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.817052 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.817151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.817184 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.817229 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
W20260619 12:03:08.817333 17216 tablet.cc:2369] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:03:08.817809 17169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8c5705ad94ca09a37605f2bbcfe30" candidate_uuid: "3bd1829110584f3cba46edc4ba528933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15af04dd98547a9b9daa7135d9842be" is_pre_election: true
I20260619 12:03:08.817808 16899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8c5705ad94ca09a37605f2bbcfe30" candidate_uuid: "3bd1829110584f3cba46edc4ba528933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b85c7fe8004f5cbd879c10569a7676" is_pre_election: true
I20260619 12:03:08.817986 17169 raft_consensus.cc:2454] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd1829110584f3cba46edc4ba528933 in term 0.
I20260619 12:03:08.817986 16899 raft_consensus.cc:2454] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd1829110584f3cba46edc4ba528933 in term 0.
I20260619 12:03:08.818344 16971 leader_election.cc:302] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd1829110584f3cba46edc4ba528933, b15af04dd98547a9b9daa7135d9842be; no voters:
I20260619 12:03:08.818512 17229 raft_consensus.cc:2781] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:03:08.818584 17229 raft_consensus.cc:482] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:03:08.818609 17229 raft_consensus.cc:3037] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:08.819492 17229 raft_consensus.cc:504] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.819701 17229 leader_election.cc:288] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [CANDIDATE]: Term 1 election: Requested vote from peers b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203), 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193:43585)
I20260619 12:03:08.819900 17169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8c5705ad94ca09a37605f2bbcfe30" candidate_uuid: "3bd1829110584f3cba46edc4ba528933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15af04dd98547a9b9daa7135d9842be"
I20260619 12:03:08.819976 16899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8c5705ad94ca09a37605f2bbcfe30" candidate_uuid: "3bd1829110584f3cba46edc4ba528933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b85c7fe8004f5cbd879c10569a7676"
I20260619 12:03:08.820046 17169 raft_consensus.cc:3037] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:08.820076 16899 raft_consensus.cc:3037] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:08.821282 17169 raft_consensus.cc:2454] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd1829110584f3cba46edc4ba528933 in term 1.
I20260619 12:03:08.821282 16899 raft_consensus.cc:2454] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd1829110584f3cba46edc4ba528933 in term 1.
I20260619 12:03:08.821553 16972 leader_election.cc:302] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676, 3bd1829110584f3cba46edc4ba528933; no voters:
I20260619 12:03:08.821817 17229 raft_consensus.cc:2781] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:08.822082 17229 raft_consensus.cc:686] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd1829110584f3cba46edc4ba528933, State: Running, Role: LEADER
I20260619 12:03:08.822177 17229 consensus_queue.cc:237] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:08.823486 16757 catalog_manager.cc:5495] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd1829110584f3cba46edc4ba528933 (127.0.203.194). New cstate: current_term: 1 leader_uuid: "3bd1829110584f3cba46edc4ba528933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } health_report { overall_health: UNKNOWN } } }
I20260619 12:03:08.850984 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:08.847921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.203.194:37103)
0619 12:03:08.848022 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:08.848025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:08.848033 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:03:08.848102 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:08.848103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:08.848125 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:08.848216 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:08.848365 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.849002 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.849866 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:08.850090 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:08.850193 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:08.850422 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:08.850593 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:03:08.850716 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:08.850768 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 12:03:08.850791 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:03:08.853446 17169 raft_consensus.cc:1261] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Refusing update from remote peer 3bd1829110584f3cba46edc4ba528933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:08.853449 16899 raft_consensus.cc:1261] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Refusing update from remote peer 3bd1829110584f3cba46edc4ba528933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:08.853935 17229 consensus_queue.cc:1035] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:08.854140 17233 consensus_queue.cc:1035] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:08.857195 17239 mvcc.cc:205] Tried to move back new op lower bound from 7298541931939975168 to 7298541931817316352. Current Snapshot: MvccSnapshot[applied={T|T < 7298541931939975168}]
I20260619 12:03:08.859918 16757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:08.860031 16757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:08.860981 16757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Sending DeleteTablet for 3 replicas of tablet d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:08.861342 17017 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f8c5705ad94ca09a37605f2bbcfe30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:08 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:48784
I20260619 12:03:08.861450 17149 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f8c5705ad94ca09a37605f2bbcfe30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:08 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:58084
I20260619 12:03:08.861692 16879 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f8c5705ad94ca09a37605f2bbcfe30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:08 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:51742
I20260619 12:03:08.861720 17245 tablet_replica.cc:330] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: stopping tablet replica
I20260619 12:03:08.861867 17245 raft_consensus.cc:2229] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:03:08.862049 17247 tablet_replica.cc:330] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: stopping tablet replica
I20260619 12:03:08.862056 17245 raft_consensus.cc:2258] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:08.862143 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 16795
I20260619 12:03:08.862244 17247 raft_consensus.cc:2229] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:08.862355 17247 raft_consensus.cc:2258] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:08.863257 17245 ts_tablet_manager.cc:1905] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:08.865803 17245 ts_tablet_manager.cc:1918] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:08.865913 17245 log.cc:1196] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:08.866267 17245 ts_tablet_manager.cc:1939] T d9f8c5705ad94ca09a37605f2bbcfe30 P 3bd1829110584f3cba46edc4ba528933: Deleting consensus metadata
I20260619 12:03:08.866990 16742 catalog_manager.cc:4841] TS 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103): tablet d9f8c5705ad94ca09a37605f2bbcfe30 (table test-table [id=e8768fb73c3e443d90fa86244445b167]) successfully deleted
W20260619 12:03:08.870170 16743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:43585 (error 108)
I20260619 12:03:08.870476 17246 tablet_replica.cc:330] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: stopping tablet replica
I20260619 12:03:08.870724 17246 raft_consensus.cc:2229] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 12:03:08.870810 17221 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:03:08.870669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:43585 (local address 127.0.0.1:51748)
0619 12:03:08.870730 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:08.870763 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:43585: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 12:03:08.870990 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 16950
I20260619 12:03:08.871091 17246 raft_consensus.cc:2258] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 12:03:08.870950 16743 catalog_manager.cc:4584] TS 20b85c7fe8004f5cbd879c10569a7676 (127.0.203.193:43585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9f8c5705ad94ca09a37605f2bbcfe30: Network error: Client connection negotiation failed: client connection to 127.0.203.193:43585: connect: Connection reset by peer (error 104)
I20260619 12:03:08.872550 17246 ts_tablet_manager.cc:1905] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:08.874753 17246 ts_tablet_manager.cc:1918] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:08.874838 17246 log.cc:1196] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:08.875205 17246 ts_tablet_manager.cc:1939] T d9f8c5705ad94ca09a37605f2bbcfe30 P b15af04dd98547a9b9daa7135d9842be: Deleting consensus metadata
I20260619 12:03:08.875769 16742 catalog_manager.cc:4841] TS b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203): tablet d9f8c5705ad94ca09a37605f2bbcfe30 (table test-table [id=e8768fb73c3e443d90fa86244445b167]) successfully deleted
I20260619 12:03:08.878490 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 17086
I20260619 12:03:08.885383 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 16724
I20260619 12:03:28.892535 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44921
--webserver_interface=127.0.203.254
--webserver_port=42595
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:28.986853 17249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:28.987136 17249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:28.987164 17249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:28.988566 17249 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:03:28.988719 17249 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:03:28.988749 17249 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:03:28.988770 17249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:28.988806 17249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:28.988848 17249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:28.988867 17249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:28.988904 17249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:28.991775 17249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=42595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:28.992292 17249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:28.992733 17249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:29.001099 17257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:29.001199 17254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:29.001278 17249 server_base.cc:1034] running on GCE node
W20260619 12:03:29.001283 17255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:29.002012 17249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:29.002393 17249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:29.003615 17249 hybrid_clock.cc:648] HybridClock initialized: now 1781870609003568 us; error 73 us; skew 500 ppm
Jun 19 12:03:29 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:29.006705 17249 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:03:29.007264 17249 webserver.cc:460] Webserver started at http://127.0.203.254:42595/ using document root <none> and password file <none>
I20260619 12:03:29.007535 17249 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:29.007591 17249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:29.009342 17249 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:29.010406 17264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.010735 17249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:29.010839 17249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "6b28349953974e949a83844850ccbe63"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9241b279fcdbc3a6e3fe44114d0d7e1a"
server_key_iv: "69f7a76a29cb48adb0d138b4da7903b5"
server_key_version: "encryptionkey@0"
I20260619 12:03:29.011279 17249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:29.050077 17249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:29.052609 17249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:29.052891 17249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:29.058116 17249 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:44921
I20260619 12:03:29.058207 17316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:44921 every 8 connection(s)
I20260619 12:03:29.058660 17249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:29.060504 17317 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:03:29.060900 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 17249
I20260619 12:03:29.061199 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:43585
--local_ip_for_outbound_sockets=127.0.203.193
--tserver_master_addrs=127.0.203.254:44921
--webserver_port=35219
--webserver_interface=127.0.203.193
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:03:29.061425 17317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Bootstrap starting.
I20260619 12:03:29.071830 17317 log.cc:824] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:29.074867 17317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:03:29.075240 17317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Bootstrap complete.
I20260619 12:03:29.077863 17317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:29.078151 17317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b28349953974e949a83844850ccbe63, State: Initialized, Role: FOLLOWER
I20260619 12:03:29.078361 17317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:29.078446 17317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:29.078501 17317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:29.078557 17317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:03:29.079705 17317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:29.079838 17317 leader_election.cc:302] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b28349953974e949a83844850ccbe63; no voters:
I20260619 12:03:29.080071 17317 leader_election.cc:288] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 12:03:29.080112 17321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:03:29.080257 17321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [term 3 LEADER]: Becoming Leader. State: Replica: 6b28349953974e949a83844850ccbe63, State: Running, Role: LEADER
I20260619 12:03:29.080353 17321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } }
I20260619 12:03:29.080423 17317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:29.080677 17321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b28349953974e949a83844850ccbe63. Latest consensus state: current_term: 3 leader_uuid: "6b28349953974e949a83844850ccbe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } } }
I20260619 12:03:29.080761 17321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:29.080690 17322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6b28349953974e949a83844850ccbe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b28349953974e949a83844850ccbe63" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 44921 } } }
I20260619 12:03:29.081259 17325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:29.081301 17322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:29.082264 17325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:29.082381 17325 catalog_manager.cc:1259] Loaded cluster ID: 521480b04ad54670b02174d04c2a682a
I20260619 12:03:29.082417 17325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:29.083591 17325 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:29.083797 17325 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Loaded TSK: 0
I20260619 12:03:29.083850 17325 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b28349953974e949a83844850ccbe63: Loaded TSK: 1
F20260619 12:03:29.084100 17325 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fed05c7d9ad google::LogMessage::Fail() at ??:0
@ 0x7fed05c8088a google::LogMessage::SendToLog() at ??:0
@ 0x7fed05c7d4de google::LogMessage::Flush() at ??:0
@ 0x7fed05c7ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fed083ea104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fed083e7d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fed0dbdb1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fed0dbdb48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fed0dbdb594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fed0dba1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fed0dbabad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fed07b4b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fed07b40954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fed089a46db start_thread at ??:0
@ 0x7fed0477b61f clone at ??:0
W20260619 12:03:29.176393 17319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:29.176781 17319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:29.176832 17319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:29.178905 17319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:29.178994 17319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:29.179041 17319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:29.181517 17319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:43585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=35219
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:29.181849 17319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:29.182142 17319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:29.188241 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:29.188380 17319 server_base.cc:1034] running on GCE node
W20260619 12:03:29.188280 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:29.188575 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:29.188905 17319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:29.189236 17319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:29.190415 17319 hybrid_clock.cc:648] HybridClock initialized: now 1781870609190387 us; error 52 us; skew 500 ppm
Jun 19 12:03:29 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:29.193459 17319 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:03:29.194032 17319 webserver.cc:460] Webserver started at http://127.0.203.193:35219/ using document root <none> and password file <none>
I20260619 12:03:29.194324 17319 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:29.194398 17319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:29.196504 17319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:29.197566 17368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.197865 17319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:29.197992 17319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "20b85c7fe8004f5cbd879c10569a7676"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "52d97ce240cdd7db00884c40aedeb674"
server_key_iv: "c2a40a2d5d048c38595037c68c1d6722"
server_key_version: "encryptionkey@0"
I20260619 12:03:29.198503 17319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:29.214594 17319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:29.217312 17319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:29.217501 17319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:29.217797 17319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:29.218348 17375 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:03:29.220538 17319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:03:29.220678 17319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260619 12:03:29.220731 17319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:03:29.221563 17319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:03:29.221628 17319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 12:03:29.221747 17375 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap starting.
I20260619 12:03:29.230422 17319 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:43585
I20260619 12:03:29.230592 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:43585 every 8 connection(s)
I20260619 12:03:29.230929 17319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:03:29.233387 17375 log.cc:824] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:29.236299 17375 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:03:29.236778 17375 tablet_bootstrap.cc:492] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Bootstrap complete.
I20260619 12:03:29.237411 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 17319
I20260619 12:03:29.237741 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:37103
--local_ip_for_outbound_sockets=127.0.203.194
--tserver_master_addrs=127.0.203.254:44921
--webserver_port=36835
--webserver_interface=127.0.203.194
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:03:29.238214 17375 ts_tablet_manager.cc:1397] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260619 12:03:29.240118 17375 raft_consensus.cc:348] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:29.240432 17375 raft_consensus.cc:729] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b85c7fe8004f5cbd879c10569a7676, State: Initialized, Role: FOLLOWER
I20260619 12:03:29.240633 17375 consensus_queue.cc:260] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:29.241000 17375 ts_tablet_manager.cc:1428] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
W20260619 12:03:29.351940 17485 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:03:29.232273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:46993)
0619 12:03:29.232497 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.232545 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:29.233211 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:29.351707 (+118496us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":141,"threads_started":1}
W20260619 12:03:29.352308 17483 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:44921 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.254:44921: Client connection negotiation failed: client connection to 127.0.203.254:44921: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 12:03:29.352192 17487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:29.352483 17487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:29.352540 17487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 12:03:29.352874 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.352740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:49273)
0619 12:03:29.352797 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.352844 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 12:03:29.353360 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.353255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:51207)
0619 12:03:29.353316 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.353342 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 12:03:29.353510 17483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:03:29.354161 17487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:29.354215 17487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:29.354247 17487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:29.355906 17487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:37103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=36835
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:29.356176 17487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:29.356446 17487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:29.362161 17493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:29.362130 17494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:29.362313 17487 server_base.cc:1034] running on GCE node
W20260619 12:03:29.362150 17496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:29.362664 17487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:29.362983 17487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:29.364163 17487 hybrid_clock.cc:648] HybridClock initialized: now 1781870609364155 us; error 60 us; skew 500 ppm
Jun 19 12:03:29 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:29.367379 17487 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:03:29.367933 17487 webserver.cc:460] Webserver started at http://127.0.203.194:36835/ using document root <none> and password file <none>
I20260619 12:03:29.368214 17487 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:29.368289 17487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:29.369928 17487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:29.370718 17503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.371028 17487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:29.371172 17487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "3bd1829110584f3cba46edc4ba528933"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "56b0e4c8a33b7ab4f74f3e51fac8664b"
server_key_iv: "effe01883d4378a9a66100926593fe38"
server_key_version: "encryptionkey@0"
I20260619 12:03:29.371625 17487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:29.392835 17487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:29.395473 17487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:29.395643 17487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:29.395927 17487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:29.396282 17487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:29.396340 17487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.396389 17487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:29.396412 17487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.404081 17487 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:37103
I20260619 12:03:29.404265 17616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:37103 every 8 connection(s)
I20260619 12:03:29.404560 17487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:03:29.405720 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 17487
I20260619 12:03:29.406008 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.405388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:44631)
0619 12:03:29.405618 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.405777 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":111,"threads_started":1}
I20260619 12:03:29.406054 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:36203
--local_ip_for_outbound_sockets=127.0.203.195
--tserver_master_addrs=127.0.203.254:44921
--webserver_port=45049
--webserver_interface=127.0.203.195
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:29.406534 17617 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:44921 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.254:44921: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
I20260619 12:03:29.406917 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.406783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:45045)
0619 12:03:29.406847 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.406883 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 12:03:29.407367 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.407249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:36893)
0619 12:03:29.407308 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.407338 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 12:03:29.407577 17617 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:03:29.502089 17620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:29.502403 17620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:29.502434 17620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:29.503743 17620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:29.503793 17620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:29.503822 17620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:29.505370 17620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:36203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=45049
--tserver_master_addrs=127.0.203.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:29.505668 17620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:29.505950 17620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:29.511921 17628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:29.511952 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:29.511921 17626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:29.512260 17620 server_base.cc:1034] running on GCE node
I20260619 12:03:29.512553 17620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:29.512871 17620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:29.514051 17620 hybrid_clock.cc:648] HybridClock initialized: now 1781870609514011 us; error 58 us; skew 500 ppm
Jun 19 12:03:29 dist-test-slave-fbq8 krb5kdc[16177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:29.517076 17620 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:03:29.517647 17620 webserver.cc:460] Webserver started at http://127.0.203.195:45049/ using document root <none> and password file <none>
I20260619 12:03:29.517918 17620 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:29.517992 17620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:29.519724 17620 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:29.520749 17635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.521028 17620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:29.521134 17620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "b15af04dd98547a9b9daa7135d9842be"
format_stamp: "Formatted at 2026-06-19 12:02:47 on dist-test-slave-fbq8"
server_key: "9605ac9f2a54668b30c13bf457d3e3f7"
server_key_iv: "1c90ff772d9895a9325bb659201dd06f"
server_key_version: "encryptionkey@0"
I20260619 12:03:29.521564 17620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:29.543792 17620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:29.546569 17620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:29.546756 17620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:29.547065 17620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:29.547441 17620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:29.547518 17620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.547595 17620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:29.547626 17620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:29.556247 17620 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:36203
I20260619 12:03:29.556341 17748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:36203 every 8 connection(s)
I20260619 12:03:29.556874 17620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:03:29.558413 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.557812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:35725)
0619 12:03:29.558096 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.558210 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":150,"threads_started":1}
W20260619 12:03:29.558913 17749 heartbeater.cc:644] Failed to heartbeat to 127.0.203.254:44921 (0 consecutive failures): Network error: Failed to ping master at 127.0.203.254:44921: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
I20260619 12:03:29.559382 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.559217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:54797)
0619 12:03:29.559294 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.559344 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:03:29.559782 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:29.559690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:46151)
0619 12:03:29.559740 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:29.559767 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 12:03:29.560102 17749 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:03:29.562960 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 17620
I20260619 12:03:30.354332 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:30.354020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:49719)
0619 12:03:30.354195 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:30.354259 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260619 12:03:30.408413 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:30.408097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:36789)
0619 12:03:30.408275 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:30.408327 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260619 12:03:30.560907 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:30.560557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:33069)
0619 12:03:30.560788 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:30.560833 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":129,"threads_started":1}
I20260619 12:03:30.582723 17756 raft_consensus.cc:482] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:30.582898 17756 raft_consensus.cc:504] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:30.583377 17756 leader_election.cc:288] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103), b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
Jun 19 12:03:30 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.194@KRBTEST.COM
Jun 19 12:03:30 dist-test-slave-fbq8 krb5kdc[16177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.195@KRBTEST.COM
I20260619 12:03:30.596485 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:30.583875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:60999 (local address 127.0.203.194:37103)
0619 12:03:30.584116 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:30.584131 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:30.584186 (+ 55us) server_negotiation.cc:407] Connection header received
0619 12:03:30.584911 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:30.584914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:30.584976 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:30.585151 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:30.586193 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.587115 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.588218 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.588704 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.588844 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:30.591785 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:30.591789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:30.591790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:30.591802 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:30.594188 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:30.594936 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:30.594940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:30.594941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:30.595067 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:30.595456 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:30.595457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:30.595457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:30.595754 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:30.595846 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:30.596154 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 12:03:30.596322 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
I20260619 12:03:30.596450 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:30.583701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:37103 (local address 127.0.203.193:60999)
0619 12:03:30.583767 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:30.583781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:30.583849 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:30.585204 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:30.585208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:30.585777 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:30.586042 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:30.586050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.587229 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.587232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:30.588100 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:30.588103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.588812 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.588814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:30.588938 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:30.589725 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:30.589730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:30.591594 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:30.594318 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:30.594327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:30.594333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:30.594822 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:30.595175 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:30.595177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:30.595177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:30.595368 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:30.595871 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:30.595872 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:30.595943 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 12:03:30.596067 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 12:03:30.596301 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":91}
I20260619 12:03:30.596733 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:30.584026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:52813 (local address 127.0.203.195:36203)
0619 12:03:30.584163 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:30.584195 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:30.584240 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:03:30.584918 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:30.584921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:30.584981 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:30.585184 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:30.586181 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.587094 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.588212 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.588632 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.588823 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:30.592123 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:30.592128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:30.592128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:30.592144 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:30.594222 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:30.594955 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:30.594959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:30.594959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:30.595119 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:30.595442 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:30.595443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:30.595443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:30.595689 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:30.595752 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:30.596410 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 12:03:30.596581 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260619 12:03:30.597054 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:30.583883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:36203 (local address 127.0.203.193:52813)
0619 12:03:30.584011 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:30.584025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:30.584076 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:30.585203 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:30.585207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:30.585779 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:30.586041 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:30.586048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.587217 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.587218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:30.588101 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:30.588103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:30.588790 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:30.588792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:30.588950 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:30.589701 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:30.589711 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:30.591962 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:30.594318 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:30.594327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:30.594329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:30.594822 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:30.595305 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:30.595306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:30.595306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:30.595381 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:30.595799 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:30.595807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:30.595955 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 12:03:30.596096 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 12:03:30.596414 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":197,"thread_start_us":63,"threads_started":1}
W20260619 12:03:30.597585 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
W20260619 12:03:30.597819 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:30.597877 17370 leader_election.cc:302] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676; no voters: 3bd1829110584f3cba46edc4ba528933, b15af04dd98547a9b9daa7135d9842be
I20260619 12:03:30.598073 17756 raft_consensus.cc:2726] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:03:31.355520 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:31.355166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:57291)
0619 12:03:31.355377 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:31.355452 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
I20260619 12:03:31.409408 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:31.409044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:47057)
0619 12:03:31.409285 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:31.409355 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":66,"threads_started":1}
I20260619 12:03:31.561764 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:31.561494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:35937)
0619 12:03:31.561665 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:31.561712 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260619 12:03:32.356392 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:32.356117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:60303)
0619 12:03:32.356274 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:32.356323 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260619 12:03:32.410261 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:32.409951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:55025)
0619 12:03:32.410160 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:32.410208 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":136,"threads_started":1}
I20260619 12:03:32.535915 17765 raft_consensus.cc:482] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:32.536180 17765 raft_consensus.cc:504] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:32.536413 17765 leader_election.cc:288] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103), b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
W20260619 12:03:32.537070 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
W20260619 12:03:32.537161 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:32.537180 17370 leader_election.cc:302] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676; no voters: 3bd1829110584f3cba46edc4ba528933, b15af04dd98547a9b9daa7135d9842be
I20260619 12:03:32.537262 17765 raft_consensus.cc:2726] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:03:32.562711 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:32.562442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:44291)
0619 12:03:32.562607 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:32.562648 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260619 12:03:33.357380 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:33.357050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:49525)
0619 12:03:33.357259 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:33.357315 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260619 12:03:33.411202 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:33.410928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:51705)
0619 12:03:33.411086 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:33.411143 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260619 12:03:33.563834 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:33.563501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:58117)
0619 12:03:33.563707 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:33.563770 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260619 12:03:34.358523 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:34.358243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:34999)
0619 12:03:34.358423 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:34.358469 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260619 12:03:34.412168 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:34.411852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:50359)
0619 12:03:34.412060 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:34.412109 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
I20260619 12:03:34.564908 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:34.564447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:41743)
0619 12:03:34.564783 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:34.564848 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260619 12:03:34.841149 17773 raft_consensus.cc:482] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:34.841291 17773 raft_consensus.cc:504] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:34.841496 17773 leader_election.cc:288] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103), b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
W20260619 12:03:34.842149 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
W20260619 12:03:34.842258 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:34.842375 17370 leader_election.cc:302] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676; no voters: 3bd1829110584f3cba46edc4ba528933, b15af04dd98547a9b9daa7135d9842be
I20260619 12:03:34.842533 17773 raft_consensus.cc:2726] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:03:35.359530 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:35.359208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:50493)
0619 12:03:35.359412 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:35.359476 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260619 12:03:35.413199 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:35.412874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:48115)
0619 12:03:35.413074 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:35.413139 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260619 12:03:35.565924 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:35.565517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:60453)
0619 12:03:35.565785 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:35.565848 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260619 12:03:36.360463 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:36.360209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:48339)
0619 12:03:36.360352 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:36.360399 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260619 12:03:36.414252 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:36.413918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:50587)
0619 12:03:36.414125 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:36.414188 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260619 12:03:36.566845 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:36.566559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:44199)
0619 12:03:36.566738 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:36.566793 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260619 12:03:37.361409 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:37.361145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:51489)
0619 12:03:37.361300 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:37.361345 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260619 12:03:37.415462 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:37.415116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:43823)
0619 12:03:37.415318 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:37.415374 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260619 12:03:37.567763 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:37.567459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:49437)
0619 12:03:37.567655 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:37.567712 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260619 12:03:38.026885 17783 raft_consensus.cc:482] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:38.027030 17783 raft_consensus.cc:504] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:38.027315 17783 leader_election.cc:288] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103), b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
W20260619 12:03:38.027971 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
W20260619 12:03:38.028110 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:38.028133 17370 leader_election.cc:302] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676; no voters: 3bd1829110584f3cba46edc4ba528933, b15af04dd98547a9b9daa7135d9842be
I20260619 12:03:38.028292 17783 raft_consensus.cc:2726] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:03:38.362545 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:38.362077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:39505)
0619 12:03:38.362272 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:38.362472 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
I20260619 12:03:38.416309 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:38.416016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:36065)
0619 12:03:38.416201 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:38.416263 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260619 12:03:38.568719 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:38.568369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:39611)
0619 12:03:38.568534 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:38.568610 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260619 12:03:39.363415 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:39.363114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:43517)
0619 12:03:39.363307 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:39.363363 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260619 12:03:39.417339 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:39.416975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:43347)
0619 12:03:39.417174 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:39.417275 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260619 12:03:39.569538 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:39.569243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:39965)
0619 12:03:39.569422 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:39.569478 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260619 12:03:40.364338 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:40.364015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:34467)
0619 12:03:40.364224 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:40.364286 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260619 12:03:40.418319 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:40.418037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:56329)
0619 12:03:40.418198 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:40.418248 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260619 12:03:40.570550 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:40.570223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:43937)
0619 12:03:40.570437 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:40.570498 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260619 12:03:40.865473 17793 raft_consensus.cc:482] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:40.865749 17793 raft_consensus.cc:504] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd1829110584f3cba46edc4ba528933" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 37103 } } peers { permanent_uuid: "b15af04dd98547a9b9daa7135d9842be" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 36203 } } peers { permanent_uuid: "20b85c7fe8004f5cbd879c10569a7676" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 43585 } }
I20260619 12:03:40.865980 17793 leader_election.cc:288] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103), b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203)
W20260619 12:03:40.866715 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bd1829110584f3cba46edc4ba528933 (127.0.203.194:37103): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
W20260619 12:03:40.866822 17370 leader_election.cc:341] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b15af04dd98547a9b9daa7135d9842be (127.0.203.195:36203): Not found: Tablet not found: d9f8c5705ad94ca09a37605f2bbcfe30
I20260619 12:03:40.866847 17370 leader_election.cc:302] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b85c7fe8004f5cbd879c10569a7676; no voters: 3bd1829110584f3cba46edc4ba528933, b15af04dd98547a9b9daa7135d9842be
I20260619 12:03:40.866920 17793 raft_consensus.cc:2726] T d9f8c5705ad94ca09a37605f2bbcfe30 P 20b85c7fe8004f5cbd879c10569a7676 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:03:41.365301 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:41.364988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:51573)
0619 12:03:41.365163 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:41.365238 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260619 12:03:41.419243 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:41.418946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:36333)
0619 12:03:41.419139 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:41.419192 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260619 12:03:41.571465 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:41.571188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:56353)
0619 12:03:41.571351 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:41.571401 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260619 12:03:42.366269 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:42.365938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:37683)
0619 12:03:42.366126 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:42.366196 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260619 12:03:42.420258 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:42.419969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:55355)
0619 12:03:42.420160 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:42.420209 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260619 12:03:42.572423 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:42.572068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:46697)
0619 12:03:42.572290 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:42.572361 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260619 12:03:43.367173 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:43.366893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:58331)
0619 12:03:43.367061 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:43.367106 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260619 12:03:43.421401 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:43.420981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:47413)
0619 12:03:43.421168 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:43.421338 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260619 12:03:43.573590 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:43.573225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:49451)
0619 12:03:43.573436 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:43.573501 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260619 12:03:44.368090 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:44.367764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.193:35755)
0619 12:03:44.367958 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:44.368022 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260619 12:03:44.422489 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:44.422209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.194:46359)
0619 12:03:44.422375 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:44.422429 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260619 12:03:44.565631 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 17319
I20260619 12:03:44.572088 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 17487
I20260619 12:03:44.574668 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:44.574406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:44921 (local address 127.0.203.195:47015)
0619 12:03:44.574566 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:44.574603 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.254:44921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260619 12:03:44.577505 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 17620
2026-06-19T12:03:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58717 ms)
[----------] 30 tests from SecurityITest (244551 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:44 dist-test-slave-fbq8 krb5kdc[17808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:44 dist-test-slave-fbq8 krb5kdc[17808](info): set up 2 sockets
Jun 19 12:03:44 dist-test-slave-fbq8 krb5kdc[17808](info): commencing operation
krb5kdc: starting...
W20260619 12:03:45.640556 815 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:03:45 dist-test-slave-fbq8 krb5kdc[17808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:03:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:03:45.775712 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33639
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:33639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:45.869278 17821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:45.869621 17821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:45.869668 17821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:45.871047 17821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:45.871127 17821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:45.871157 17821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:45.871192 17821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:45.871222 17821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:45.873078 17821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:33639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:45.873415 17821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:45.873700 17821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:45.880152 17829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:45.880152 17826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:45.880282 17821 server_base.cc:1034] running on GCE node
W20260619 12:03:45.880164 17827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:45.880932 17821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:45.881302 17821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:45.882500 17821 hybrid_clock.cc:648] HybridClock initialized: now 1781870625882458 us; error 73 us; skew 500 ppm
Jun 19 12:03:45 dist-test-slave-fbq8 krb5kdc[17808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:45.885686 17821 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:03:45.886260 17821 webserver.cc:460] Webserver started at http://127.0.203.254:43451/ using document root <none> and password file <none>
I20260619 12:03:45.886533 17821 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:45.886608 17821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:45.886766 17821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:45.887995 17821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0"
format_stamp: "Formatted at 2026-06-19 12:03:45 on dist-test-slave-fbq8"
server_key: "2b34162a36de6cfb34f9241891bc70a1"
server_key_iv: "b74e3cfc5294595bd6ddef0155c6a9ae"
server_key_version: "encryptionkey@0"
I20260619 12:03:45.888440 17821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0"
format_stamp: "Formatted at 2026-06-19 12:03:45 on dist-test-slave-fbq8"
server_key: "2b34162a36de6cfb34f9241891bc70a1"
server_key_iv: "b74e3cfc5294595bd6ddef0155c6a9ae"
server_key_version: "encryptionkey@0"
I20260619 12:03:45.890098 17821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:45.891039 17836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:45.891423 17821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:45.891561 17821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0"
format_stamp: "Formatted at 2026-06-19 12:03:45 on dist-test-slave-fbq8"
server_key: "2b34162a36de6cfb34f9241891bc70a1"
server_key_iv: "b74e3cfc5294595bd6ddef0155c6a9ae"
server_key_version: "encryptionkey@0"
I20260619 12:03:45.891676 17821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:45.905431 17821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:45.908427 17821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:45.908628 17821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:45.914352 17821 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:33639
I20260619 12:03:45.914394 17888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:33639 every 8 connection(s)
I20260619 12:03:45.914824 17821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:45.915839 17889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:45.918839 17889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0: Bootstrap starting.
I20260619 12:03:45.919510 17889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:45.919833 17889 log.cc:824] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:45.921132 17889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0: No bootstrap required, opened a new log
I20260619 12:03:45.921973 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 17821
I20260619 12:03:45.922099 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:45.922358 815 external_mini_cluster.cc:1428] Setting key 011e3c001cf446d11ed30e32bb965a8b
I20260619 12:03:45.923220 17889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33639 } }
I20260619 12:03:45.923499 17889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:45.923575 17889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043cb5a5c2ce4f6eb95de9a6eefad6d0, State: Initialized, Role: FOLLOWER
I20260619 12:03:45.923760 17889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33639 } }
I20260619 12:03:45.923869 17889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:45.923913 17889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:45.923969 17889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:45.925168 17889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33639 } }
I20260619 12:03:45.925294 17889 leader_election.cc:302] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043cb5a5c2ce4f6eb95de9a6eefad6d0; no voters:
I20260619 12:03:45.925479 17889 leader_election.cc:288] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:45.925587 17894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:45.925794 17889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:45.925832 17894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [term 1 LEADER]: Becoming Leader. State: Replica: 043cb5a5c2ce4f6eb95de9a6eefad6d0, State: Running, Role: LEADER
I20260619 12:03:45.925952 17894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33639 } }
I20260619 12:03:45.926502 17894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043cb5a5c2ce4f6eb95de9a6eefad6d0. Latest consensus state: current_term: 1 leader_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33639 } } }
I20260619 12:03:45.926622 17894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:45.926522 17895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043cb5a5c2ce4f6eb95de9a6eefad6d0" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33639 } } }
I20260619 12:03:45.926766 17895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0 [sys.catalog]: This master's current role is: LEADER
W20260619 12:03:45.928197 17909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:03:45.928309 17909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:03:45.928424 17903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:45.928611 17903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 12:03:45 dist-test-slave-fbq8 krb5kdc[17808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:45.932142 17903 catalog_manager.cc:1347] Generated new cluster ID: 8defbede49b74211bee438f88bbf3c9f
I20260619 12:03:45.932222 17903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:45.940728 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:45.923228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.0.203.254:33639)
0619 12:03:45.923462 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:45.923494 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:45.923542 (+ 48us) server_negotiation.cc:407] Connection header received
0619 12:03:45.923732 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:45.923741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:45.923805 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:45.923987 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:45.924915 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:45.926364 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:45.927777 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:45.928207 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:45.928341 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:45.931651 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:45.931655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:45.931655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:45.931667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:45.938595 (+ 6928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:45.939197 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:45.939201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:45.939201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:45.939346 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:45.939602 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:45.939602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:45.939603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:45.939929 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:45.940095 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:45.940270 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 12:03:45.940430 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":131,"threads_started":1}
I20260619 12:03:45.989185 17903 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:45.989938 17903 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:46.002521 17903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0: Generated new TSK 0
I20260619 12:03:46.002767 17903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:03:46.045109 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33639
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:46.134135 17915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:46.134390 17915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:46.134430 17915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:46.135602 17915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:46.135689 17915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:46.135735 17915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:46.137333 17915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:46.137625 17915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:46.137869 17915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:46.143617 17920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:46.143610 17921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:46.143587 17923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:46.143651 17915 server_base.cc:1034] running on GCE node
I20260619 12:03:46.144210 17915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:46.144558 17915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:46.145766 17915 hybrid_clock.cc:648] HybridClock initialized: now 1781870626145733 us; error 49 us; skew 500 ppm
Jun 19 12:03:46 dist-test-slave-fbq8 krb5kdc[17808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:46.148799 17915 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:03:46.149351 17915 webserver.cc:460] Webserver started at http://127.0.203.193:42587/ using document root <none> and password file <none>
I20260619 12:03:46.149618 17915 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:46.149667 17915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:46.149808 17915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:46.151007 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "e74f8ec371fe4844a525d0e6d9a58315"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "7ff81f246c89d1b95a6f11f06833e639"
server_key_iv: "831a73ddd60ac852f3e005e383f7215f"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.151403 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "e74f8ec371fe4844a525d0e6d9a58315"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "7ff81f246c89d1b95a6f11f06833e639"
server_key_iv: "831a73ddd60ac852f3e005e383f7215f"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.153004 17915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:03:46.153924 17930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.154222 17915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:46.154306 17915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "e74f8ec371fe4844a525d0e6d9a58315"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "7ff81f246c89d1b95a6f11f06833e639"
server_key_iv: "831a73ddd60ac852f3e005e383f7215f"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.154419 17915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:46.172122 17915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:46.174564 17915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:46.174688 17915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:46.174988 17915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:46.175336 17915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:46.175408 17915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.175454 17915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:46.175482 17915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.182838 17915 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:41785
I20260619 12:03:46.182961 18043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:41785 every 8 connection(s)
I20260619 12:03:46.183286 17915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
Jun 19 12:03:46 dist-test-slave-fbq8 krb5kdc[17808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:46.192040 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 17915
I20260619 12:03:46.192186 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:03:46.192400 815 external_mini_cluster.cc:1428] Setting key 55d2350e46a3fb9370453bda4219cc13
I20260619 12:03:46.194440 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.184161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:49023 (local address 127.0.203.254:33639)
0619 12:03:46.184229 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.184232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.184528 (+ 296us) server_negotiation.cc:407] Connection header received
0619 12:03:46.185533 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.185535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.185551 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.185640 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:46.186328 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.186880 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.188062 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.188330 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.188425 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.190832 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:46.190834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:46.190834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:46.190843 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:46.192375 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.192915 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.192916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.192917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.192981 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.193446 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.193447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.193447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.193582 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:46.193680 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.194118 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.194273 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:03:46.194525 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.184100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33639 (local address 127.0.203.193:49023)
0619 12:03:46.184290 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.184334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.185377 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.185723 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.185727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.185991 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:46.186205 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.186208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.187025 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.187026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.187943 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.187945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.188442 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.188442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.188565 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.189206 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:46.189223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:46.190727 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:46.192473 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:46.192480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:46.192482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:46.192820 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:46.193086 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:46.193088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:46.193089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:46.193374 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:46.193674 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:46.193683 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:46.193855 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.194006 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.194232 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260619 12:03:46.196143 18044 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33639
I20260619 12:03:46.196282 18044 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:46.196511 18044 heartbeater.cc:511] Master 127.0.203.254:33639 requested a full tablet report, sending...
I20260619 12:03:46.197218 17853 ts_manager.cc:194] Registered new tserver with Master: e74f8ec371fe4844a525d0e6d9a58315 (127.0.203.193:41785)
I20260619 12:03:46.197950 17853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:49023
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:03:46.223241 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33639
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:46.310667 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:46.310927 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:46.310971 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:46.312209 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:46.312299 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:46.312350 18049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:46.314174 18049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:46.314417 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:46.314621 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:46.320699 18054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:46.320717 18057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:46.320720 18055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:46.321043 18049 server_base.cc:1034] running on GCE node
I20260619 12:03:46.321337 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:46.321663 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:46.322873 18049 hybrid_clock.cc:648] HybridClock initialized: now 1781870626322870 us; error 61 us; skew 500 ppm
Jun 19 12:03:46 dist-test-slave-fbq8 krb5kdc[17808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:46.326799 18049 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:03:46.327459 18049 webserver.cc:460] Webserver started at http://127.0.203.194:33395/ using document root <none> and password file <none>
I20260619 12:03:46.327766 18049 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:46.327826 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:46.328054 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:46.329530 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "448357f557a24b14acea991f8b3a526d"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "1f70af24ae858fb2f1433548736ac5d8"
server_key_iv: "c3a0cfbcff8b48577abaa95fa27edd34"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.329964 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "448357f557a24b14acea991f8b3a526d"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "1f70af24ae858fb2f1433548736ac5d8"
server_key_iv: "c3a0cfbcff8b48577abaa95fa27edd34"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.331580 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:03:46.332578 18064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.332890 18049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:46.333009 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "448357f557a24b14acea991f8b3a526d"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "1f70af24ae858fb2f1433548736ac5d8"
server_key_iv: "c3a0cfbcff8b48577abaa95fa27edd34"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.333102 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:46.348455 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:46.351228 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:46.351405 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:46.351702 18049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:46.352059 18049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:46.352099 18049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.352123 18049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:46.352136 18049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.360812 18049 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:36187
I20260619 12:03:46.360939 18177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:36187 every 8 connection(s)
I20260619 12:03:46.361277 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:03:46 dist-test-slave-fbq8 krb5kdc[17808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:46.370211 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18049
I20260619 12:03:46.370352 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:03:46.370585 815 external_mini_cluster.cc:1428] Setting key 355a850e84afa598db691f625940eff2
I20260619 12:03:46.372768 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.362279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:53797 (local address 127.0.203.254:33639)
0619 12:03:46.362350 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.362352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.362579 (+ 227us) server_negotiation.cc:407] Connection header received
0619 12:03:46.363625 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.363626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.363638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.363718 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:46.364412 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.365022 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.366142 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.366343 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.366405 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.369089 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:46.369091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:46.369091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:46.369099 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:46.370572 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.371257 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.371258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.371259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.371329 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.371748 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.371751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.371752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.372025 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:46.372125 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.372451 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.372589 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:03:46.372865 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.362156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33639 (local address 127.0.203.194:53797)
0619 12:03:46.362398 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.362447 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.363506 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.363790 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.363794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.364090 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:46.364318 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.364322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.365131 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.365132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.366054 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.366057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.366440 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.366441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.366546 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.367163 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:46.367173 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:46.368949 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:46.370674 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:46.370690 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:46.370692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:46.371160 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:46.371469 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:46.371470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:46.371470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:46.371637 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:46.372138 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:46.372144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:46.372302 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.372402 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.372559 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":132,"threads_started":1}
I20260619 12:03:46.373649 18178 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33639
I20260619 12:03:46.373850 18178 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:46.374053 18178 heartbeater.cc:511] Master 127.0.203.254:33639 requested a full tablet report, sending...
I20260619 12:03:46.374445 17853 ts_manager.cc:194] Registered new tserver with Master: 448357f557a24b14acea991f8b3a526d (127.0.203.194:36187)
I20260619 12:03:46.374876 17853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:53797
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:03:46.405649 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33639
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:46.496620 18183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:46.496927 18183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:46.496953 18183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:46.498126 18183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:46.498163 18183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:46.498193 18183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:46.499696 18183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:46.499920 18183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:46.500128 18183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:46.506215 18191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:46.506219 18188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:46.506210 18189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:46.506210 18183 server_base.cc:1034] running on GCE node
I20260619 12:03:46.506779 18183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:46.507090 18183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:46.508277 18183 hybrid_clock.cc:648] HybridClock initialized: now 1781870626508266 us; error 35 us; skew 500 ppm
Jun 19 12:03:46 dist-test-slave-fbq8 krb5kdc[17808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:46.511301 18183 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:03:46.511811 18183 webserver.cc:460] Webserver started at http://127.0.203.195:44079/ using document root <none> and password file <none>
I20260619 12:03:46.512065 18183 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:46.512116 18183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:46.512269 18183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:46.513451 18183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "836d08f8f83a4d4aba9b6ca96cf22b04"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "1c77f61f98663beae1e979d07af07355"
server_key_iv: "7018271d6f129e9dea51954c246e8201"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.513823 18183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "836d08f8f83a4d4aba9b6ca96cf22b04"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "1c77f61f98663beae1e979d07af07355"
server_key_iv: "7018271d6f129e9dea51954c246e8201"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.515251 18183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:03:46.516129 18198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.516381 18183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:46.516510 18183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "836d08f8f83a4d4aba9b6ca96cf22b04"
format_stamp: "Formatted at 2026-06-19 12:03:46 on dist-test-slave-fbq8"
server_key: "1c77f61f98663beae1e979d07af07355"
server_key_iv: "7018271d6f129e9dea51954c246e8201"
server_key_version: "encryptionkey@0"
I20260619 12:03:46.516620 18183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:46.531646 18183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:46.534301 18183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:46.534471 18183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:46.534777 18183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:46.535151 18183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:46.535220 18183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.535267 18183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:46.535295 18183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:46.542589 18183 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:40793
I20260619 12:03:46.542743 18311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:40793 every 8 connection(s)
I20260619 12:03:46.543063 18183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:03:46 dist-test-slave-fbq8 krb5kdc[17808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:46.552129 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18183
I20260619 12:03:46.552273 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:03:46.552498 815 external_mini_cluster.cc:1428] Setting key 365ddc35b24c11c0cbc353fa50da597f
I20260619 12:03:46.554522 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.544059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:51683 (local address 127.0.203.254:33639)
0619 12:03:46.544179 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.544182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.544410 (+ 228us) server_negotiation.cc:407] Connection header received
0619 12:03:46.545504 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.545506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.545518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.545586 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:46.546298 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.546847 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.548080 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.548400 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.548482 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.551151 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:46.551153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:46.551153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:46.551161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:46.552819 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.553393 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.553394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.553395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.553466 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.553890 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.553891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.553891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.554015 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:46.554070 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.554370 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.554447 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 12:03:46.554728 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.543980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33639 (local address 127.0.203.195:51683)
0619 12:03:46.544229 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.544273 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.545366 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.545649 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.545653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.545967 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:46.546187 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.546191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.546996 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.546998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.547917 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.547920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.548560 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.548562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.548775 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.549405 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:46.549415 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:46.551035 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:46.552905 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:46.552913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:46.552915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:46.553289 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:46.553598 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:46.553599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:46.553599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:46.553792 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:46.554071 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:46.554075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:46.554234 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.554326 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.554461 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":123,"threads_started":1}
I20260619 12:03:46.555276 18312 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33639
I20260619 12:03:46.555389 18312 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:46.555581 18312 heartbeater.cc:511] Master 127.0.203.254:33639 requested a full tablet report, sending...
I20260619 12:03:46.556012 17853 ts_manager.cc:194] Registered new tserver with Master: 836d08f8f83a4d4aba9b6ca96cf22b04 (127.0.203.195:40793)
I20260619 12:03:46.556438 17853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:51683
I20260619 12:03:46.564976 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:03:46.573117 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.566472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.0.203.254:33639)
0619 12:03:46.566570 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.566572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.566758 (+ 186us) server_negotiation.cc:407] Connection header received
0619 12:03:46.566786 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.566787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.566799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.566858 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:46.567437 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.567998 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.568841 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.569036 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.569107 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.570318 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:46.570319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:46.570320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:46.570326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:46.571907 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.572247 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.572248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.572249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.572305 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:46.572490 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:46.572491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:46.572491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:46.572722 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:46.572846 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.572941 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.573005 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:03:46.575342 17853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:03:46.577606 17853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:03:46.586005 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.580801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:40793 (local address 127.0.0.1:56534)
0619 12:03:46.580991 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.581053 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.581130 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.581670 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.581671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.581672 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:46.581897 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.581900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.583243 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.583244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.584664 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.584666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.585653 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.585654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.585747 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.585765 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.585817 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.585891 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":204,"thread_start_us":75,"threads_started":1}
I20260619 12:03:46.586189 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.580801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41785 (local address 127.0.0.1:40408)
0619 12:03:46.580996 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.581051 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.581120 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.581667 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.581670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.581672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:46.581788 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.581792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.583256 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.583257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.584738 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.584741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.585848 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.585849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.585928 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.585936 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.585968 (+ 32us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.586038 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":106,"threads_started":1}
I20260619 12:03:46.586299 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.581023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:36187 (local address 127.0.0.1:41286)
0619 12:03:46.581256 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.581266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.581305 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.581784 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.581786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.581789 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:46.581890 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.581892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.582925 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.582926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.584662 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.584666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.586052 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.586054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.586120 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.586125 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.586181 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.586238 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":39,"thread_start_us":127,"threads_started":1}
I20260619 12:03:46.586548 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.581345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.0.203.194:36187)
0619 12:03:46.581602 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.581608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.581619 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:46.581677 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.581679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.581698 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.581786 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:46.581969 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.582804 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.584770 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.585543 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.585944 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.586024 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.586408 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.586463 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":122,"threads_started":1}
I20260619 12:03:46.586745 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.581190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.0.203.193:41785)
0619 12:03:46.581473 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.581478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.581489 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:46.581526 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.581528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.581544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.581874 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:46.581969 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.583144 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.584844 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.585725 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.586476 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.586566 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.586597 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.586648 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":113,"threads_started":1}
I20260619 12:03:46.587203 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.580945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.0.203.195:40793)
0619 12:03:46.581452 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.581458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.581471 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:03:46.581535 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.581538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.581553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.581713 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:46.581977 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.583119 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.584776 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.585540 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.586596 (+ 1056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.587046 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.587073 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.587122 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":107,"threads_started":1}
I20260619 12:03:46.587363 18108 tablet_service.cc:1467] Processing CreateTablet for tablet ac5102e3d4844d848adac6a927a065dd (DEFAULT_TABLE table=test-table [id=2d614d329fde440883bcc5a0ad0384a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:46.587540 17978 tablet_service.cc:1467] Processing CreateTablet for tablet ac5102e3d4844d848adac6a927a065dd (DEFAULT_TABLE table=test-table [id=2d614d329fde440883bcc5a0ad0384a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:46.587809 18108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5102e3d4844d848adac6a927a065dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:46.587849 18246 tablet_service.cc:1467] Processing CreateTablet for tablet ac5102e3d4844d848adac6a927a065dd (DEFAULT_TABLE table=test-table [id=2d614d329fde440883bcc5a0ad0384a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:46.587947 17978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5102e3d4844d848adac6a927a065dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:46.588174 18246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5102e3d4844d848adac6a927a065dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:46.591815 18330 tablet_bootstrap.cc:492] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Bootstrap starting.
I20260619 12:03:46.592038 18331 tablet_bootstrap.cc:492] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: Bootstrap starting.
I20260619 12:03:46.592103 18329 tablet_bootstrap.cc:492] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Bootstrap starting.
I20260619 12:03:46.592780 18331 tablet_bootstrap.cc:654] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:46.592780 18329 tablet_bootstrap.cc:654] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:46.593348 18330 tablet_bootstrap.cc:654] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:46.593417 18331 log.cc:824] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:46.593417 18329 log.cc:824] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:46.594028 18330 log.cc:824] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:46.594303 18329 tablet_bootstrap.cc:492] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: No bootstrap required, opened a new log
I20260619 12:03:46.594372 18331 tablet_bootstrap.cc:492] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: No bootstrap required, opened a new log
I20260619 12:03:46.594385 18329 ts_tablet_manager.cc:1397] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 12:03:46.594426 18331 ts_tablet_manager.cc:1397] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 12:03:46.594874 18330 tablet_bootstrap.cc:492] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: No bootstrap required, opened a new log
I20260619 12:03:46.594969 18330 ts_tablet_manager.cc:1397] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:03:46.596550 18331 raft_consensus.cc:348] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.596550 18329 raft_consensus.cc:348] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.596796 18329 raft_consensus.cc:374] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:46.596796 18331 raft_consensus.cc:374] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:46.596850 18331 raft_consensus.cc:729] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74f8ec371fe4844a525d0e6d9a58315, State: Initialized, Role: FOLLOWER
I20260619 12:03:46.596850 18329 raft_consensus.cc:729] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836d08f8f83a4d4aba9b6ca96cf22b04, State: Initialized, Role: FOLLOWER
I20260619 12:03:46.596854 18330 raft_consensus.cc:348] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.596998 18329 consensus_queue.cc:260] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.596998 18331 consensus_queue.cc:260] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.597047 18330 raft_consensus.cc:374] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:46.597087 18330 raft_consensus.cc:729] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448357f557a24b14acea991f8b3a526d, State: Initialized, Role: FOLLOWER
I20260619 12:03:46.597218 18330 consensus_queue.cc:260] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.597319 18329 ts_tablet_manager.cc:1428] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 12:03:46.597342 18331 ts_tablet_manager.cc:1428] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 12:03:46.597460 18044 heartbeater.cc:503] Master 127.0.203.254:33639 was elected leader, sending a full tablet report...
I20260619 12:03:46.597683 18312 heartbeater.cc:503] Master 127.0.203.254:33639 was elected leader, sending a full tablet report...
I20260619 12:03:46.597896 18330 ts_tablet_manager.cc:1428] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 12:03:46.598140 18178 heartbeater.cc:503] Master 127.0.203.254:33639 was elected leader, sending a full tablet report...
W20260619 12:03:46.611925 18179 tablet.cc:2369] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:03:46.684108 18045 tablet.cc:2369] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:03:46.793819 18313 tablet.cc:2369] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:03:46.805018 18336 raft_consensus.cc:482] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:46.805179 18336 raft_consensus.cc:504] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.805509 18336 leader_election.cc:288] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e74f8ec371fe4844a525d0e6d9a58315 (127.0.203.193:41785), 448357f557a24b14acea991f8b3a526d (127.0.203.194:36187)
I20260619 12:03:46.809180 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.805893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:48933 (local address 127.0.203.193:41785)
0619 12:03:46.806048 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.806050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.806059 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:03:46.806090 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.806091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.806100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.806174 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:46.806397 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.806983 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.808153 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.808743 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.808824 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.808839 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.809064 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.809110 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 12:03:46.809181 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.805838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41785 (local address 127.0.203.195:48933)
0619 12:03:46.805908 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.805919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.805972 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.806180 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.806181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.806182 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:46.806283 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.806285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.807088 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.807089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.808050 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.808053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.808837 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.808839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.808948 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.808953 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.809036 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.809091 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:03:46.809739 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5102e3d4844d848adac6a927a065dd" candidate_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74f8ec371fe4844a525d0e6d9a58315" is_pre_election: true
I20260619 12:03:46.809896 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.806194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:36187 (local address 127.0.203.195:40269)
0619 12:03:46.806433 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.806441 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:46.806507 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:46.806662 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:46.806662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:46.806664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:46.806729 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.806730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.807600 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.807602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.808725 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:46.808728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.809602 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.809603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:46.809698 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.809701 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:46.809771 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:03:46.809817 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1,"thread_start_us":189,"threads_started":1}
I20260619 12:03:46.809885 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.806231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:40269 (local address 127.0.203.194:36187)
0619 12:03:46.806302 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.806305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.806496 (+ 191us) server_negotiation.cc:407] Connection header received
0619 12:03:46.806576 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.806577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.806587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.806658 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:46.806807 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.807476 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.808835 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.809505 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.809578 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.809589 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.809777 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.809815 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 12:03:46.809970 17998 raft_consensus.cc:2454] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836d08f8f83a4d4aba9b6ca96cf22b04 in term 0.
I20260619 12:03:46.810283 18200 leader_election.cc:302] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 836d08f8f83a4d4aba9b6ca96cf22b04, e74f8ec371fe4844a525d0e6d9a58315; no voters:
I20260619 12:03:46.810474 18336 raft_consensus.cc:2781] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:03:46.810568 18336 raft_consensus.cc:482] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:03:46.810593 18336 raft_consensus.cc:3037] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:46.810714 18132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5102e3d4844d848adac6a927a065dd" candidate_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448357f557a24b14acea991f8b3a526d" is_pre_election: true
I20260619 12:03:46.810863 18132 raft_consensus.cc:2454] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836d08f8f83a4d4aba9b6ca96cf22b04 in term 0.
I20260619 12:03:46.811681 18336 raft_consensus.cc:504] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.811841 18336 leader_election.cc:288] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [CANDIDATE]: Term 1 election: Requested vote from peers e74f8ec371fe4844a525d0e6d9a58315 (127.0.203.193:41785), 448357f557a24b14acea991f8b3a526d (127.0.203.194:36187)
I20260619 12:03:46.812052 18132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5102e3d4844d848adac6a927a065dd" candidate_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448357f557a24b14acea991f8b3a526d"
I20260619 12:03:46.812163 18132 raft_consensus.cc:3037] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:46.813158 18132 raft_consensus.cc:2454] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836d08f8f83a4d4aba9b6ca96cf22b04 in term 1.
I20260619 12:03:46.813338 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5102e3d4844d848adac6a927a065dd" candidate_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74f8ec371fe4844a525d0e6d9a58315"
I20260619 12:03:46.813413 18200 leader_election.cc:302] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448357f557a24b14acea991f8b3a526d, 836d08f8f83a4d4aba9b6ca96cf22b04; no voters:
I20260619 12:03:46.813437 17998 raft_consensus.cc:3037] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:46.813571 18336 raft_consensus.cc:2781] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:46.813823 18336 raft_consensus.cc:686] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 1 LEADER]: Becoming Leader. State: Replica: 836d08f8f83a4d4aba9b6ca96cf22b04, State: Running, Role: LEADER
I20260619 12:03:46.814141 18336 consensus_queue.cc:237] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } }
I20260619 12:03:46.814453 17998 raft_consensus.cc:2454] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836d08f8f83a4d4aba9b6ca96cf22b04 in term 1.
I20260619 12:03:46.815198 17852 catalog_manager.cc:5495] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836d08f8f83a4d4aba9b6ca96cf22b04 (127.0.203.195). New cstate: current_term: 1 leader_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836d08f8f83a4d4aba9b6ca96cf22b04" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 40793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 } health_report { overall_health: UNKNOWN } } }
I20260619 12:03:46.873649 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.870978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.0.203.195:40793)
0619 12:03:46.871078 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:46.871081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:46.871097 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:03:46.871154 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:46.871155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:46.871178 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:46.871288 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:46.871452 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.871974 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.872803 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:46.873000 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:46.873080 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:46.873261 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:46.873394 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:03:46.873498 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:46.873516 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 12:03:46.873539 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 12:03:46.876029 18132 raft_consensus.cc:1261] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 1 FOLLOWER]: Refusing update from remote peer 836d08f8f83a4d4aba9b6ca96cf22b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:46.876457 17998 raft_consensus.cc:1261] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315 [term 1 FOLLOWER]: Refusing update from remote peer 836d08f8f83a4d4aba9b6ca96cf22b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:46.876474 18339 consensus_queue.cc:1035] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448357f557a24b14acea991f8b3a526d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:46.876907 18336 consensus_queue.cc:1035] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e74f8ec371fe4844a525d0e6d9a58315" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:46.879153 18343 mvcc.cc:205] Tried to move back new op lower bound from 7298542087680393216 to 7298542087431643136. Current Snapshot: MvccSnapshot[applied={T|T < 7298542087680393216}]
I20260619 12:03:46.881814 17852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:46.881924 17852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:46.882829 17852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 043cb5a5c2ce4f6eb95de9a6eefad6d0: Sending DeleteTablet for 3 replicas of tablet ac5102e3d4844d848adac6a927a065dd
I20260619 12:03:46.883231 18246 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5102e3d4844d848adac6a927a065dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:46 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:56534
I20260619 12:03:46.883253 17978 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5102e3d4844d848adac6a927a065dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:46 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:40408
I20260619 12:03:46.883486 18108 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5102e3d4844d848adac6a927a065dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:46 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:41286
I20260619 12:03:46.883729 18352 tablet_replica.cc:330] T ac5102e3d4844d848adac6a927a065dd P e74f8ec371fe4844a525d0e6d9a58315: stopping tablet replica
I20260619 12:03:46.883874 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 17915
I20260619 12:03:46.884944 18353 tablet_replica.cc:330] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: stopping tablet replica
I20260619 12:03:46.885058 18353 raft_consensus.cc:2229] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:46.885195 18353 raft_consensus.cc:2258] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:46.885783 18353 ts_tablet_manager.cc:1905] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:46.887887 18353 ts_tablet_manager.cc:1918] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:46.887971 18353 log.cc:1196] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/ac5102e3d4844d848adac6a927a065dd
I20260619 12:03:46.888322 18353 ts_tablet_manager.cc:1939] T ac5102e3d4844d848adac6a927a065dd P 448357f557a24b14acea991f8b3a526d: Deleting consensus metadata
I20260619 12:03:46.889453 17838 catalog_manager.cc:4841] TS 448357f557a24b14acea991f8b3a526d (127.0.203.194:36187): tablet ac5102e3d4844d848adac6a927a065dd (table test-table [id=2d614d329fde440883bcc5a0ad0384a5]) successfully deleted
W20260619 12:03:46.890944 17838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:41785 (error 108)
I20260619 12:03:46.891228 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18049
I20260619 12:03:46.892042 18351 tablet_replica.cc:330] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: stopping tablet replica
I20260619 12:03:46.892161 18351 raft_consensus.cc:2229] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:03:46.892254 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:46.891414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41785 (local address 127.0.0.1:40420)
0619 12:03:46.892115 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:46.892199 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458}
I20260619 12:03:46.892283 18351 raft_consensus.cc:2258] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:46.892882 18351 ts_tablet_manager.cc:1905] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 12:03:46.893170 17838 catalog_manager.cc:4584] TS e74f8ec371fe4844a525d0e6d9a58315 (127.0.203.193:41785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac5102e3d4844d848adac6a927a065dd: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41785: connect: Connection refused (error 111)
I20260619 12:03:46.895347 18351 ts_tablet_manager.cc:1918] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:46.895443 18351 log.cc:1196] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/ac5102e3d4844d848adac6a927a065dd
I20260619 12:03:46.895740 18351 ts_tablet_manager.cc:1939] T ac5102e3d4844d848adac6a927a065dd P 836d08f8f83a4d4aba9b6ca96cf22b04: Deleting consensus metadata
I20260619 12:03:46.896302 17839 catalog_manager.cc:4841] TS 836d08f8f83a4d4aba9b6ca96cf22b04 (127.0.203.195:40793): tablet ac5102e3d4844d848adac6a927a065dd (table test-table [id=2d614d329fde440883bcc5a0ad0384a5]) successfully deleted
I20260619 12:03:46.898913 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18183
I20260619 12:03:46.905663 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 17821
2026-06-19T12:03:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2328 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T12:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:46Z Disabled control of system clock
I20260619 12:03:46.937778 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33271
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:03:47.028080 18358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:47.028338 18358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:47.028362 18358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:47.029534 18358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:47.029584 18358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:47.029598 18358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:47.029611 18358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:47.029623 18358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:47.030922 18358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:33271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:47.031157 18358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:47.031414 18358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:47.036670 18363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:47.036897 18364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.036998 18358 server_base.cc:1034] running on GCE node
W20260619 12:03:47.036761 18366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.037381 18358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:47.037716 18358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:47.038875 18358 hybrid_clock.cc:648] HybridClock initialized: now 1781870627038849 us; error 48 us; skew 500 ppm
I20260619 12:03:47.040342 18358 webserver.cc:460] Webserver started at http://127.0.203.254:46873/ using document root <none> and password file <none>
I20260619 12:03:47.040675 18358 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:47.040746 18358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:47.040884 18358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:47.042037 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "e1b3f78199c04272b497a7cbfbfa7d2c"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "f080db5c3d8432231ec7ca37524f6500"
server_key_iv: "b878207f21ddf397a0eb23d71f4d8c7e"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.042412 18358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "e1b3f78199c04272b497a7cbfbfa7d2c"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "f080db5c3d8432231ec7ca37524f6500"
server_key_iv: "b878207f21ddf397a0eb23d71f4d8c7e"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.043862 18358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 12:03:47.044759 18372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.044996 18358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:47.045116 18358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "e1b3f78199c04272b497a7cbfbfa7d2c"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "f080db5c3d8432231ec7ca37524f6500"
server_key_iv: "b878207f21ddf397a0eb23d71f4d8c7e"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.045208 18358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:47.066629 18358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:47.067013 18358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:47.067178 18358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:47.071898 18358 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:33271
I20260619 12:03:47.071966 18424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:33271 every 8 connection(s)
I20260619 12:03:47.072363 18358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:47.073558 18425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:47.075004 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18358
I20260619 12:03:47.075138 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:47.075378 815 external_mini_cluster.cc:1428] Setting key daaaf17617ae180934ede01d78654f2a
I20260619 12:03:47.076838 18425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c: Bootstrap starting.
I20260619 12:03:47.077659 18425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:47.077999 18425 log.cc:824] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:47.078758 18425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c: No bootstrap required, opened a new log
I20260619 12:03:47.081396 18425 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33271 } }
I20260619 12:03:47.081614 18425 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:47.081665 18425 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b3f78199c04272b497a7cbfbfa7d2c, State: Initialized, Role: FOLLOWER
I20260619 12:03:47.081815 18425 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33271 } }
I20260619 12:03:47.081914 18425 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:47.081959 18425 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:47.082010 18425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:47.082567 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.076068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.0.203.254:33271)
0619 12:03:47.076299 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.076308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.076340 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:03:47.077070 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.077079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.077154 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.077298 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:47.077606 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.078737 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.079989 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.080503 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.080628 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.081572 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:47.081577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:47.081577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:47.081841 (+ 264us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:47.081881 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:47.081893 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:47.082117 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.082224 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.082321 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260619 12:03:47.082731 18425 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33271 } }
I20260619 12:03:47.082854 18425 leader_election.cc:302] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b3f78199c04272b497a7cbfbfa7d2c; no voters:
I20260619 12:03:47.083040 18425 leader_election.cc:288] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:47.083084 18430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:47.083215 18430 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [term 1 LEADER]: Becoming Leader. State: Replica: e1b3f78199c04272b497a7cbfbfa7d2c, State: Running, Role: LEADER
I20260619 12:03:47.083312 18430 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33271 } }
I20260619 12:03:47.083359 18425 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:47.083674 18430 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b3f78199c04272b497a7cbfbfa7d2c. Latest consensus state: current_term: 1 leader_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33271 } } }
I20260619 12:03:47.083770 18430 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:47.083688 18432 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b3f78199c04272b497a7cbfbfa7d2c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33271 } } }
I20260619 12:03:47.084335 18432 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:47.084532 18435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:47.085100 18435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:47.086833 18435 catalog_manager.cc:1347] Generated new cluster ID: 75ee868e78b6477faf0026f5589fd6ac
I20260619 12:03:47.086911 18435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:47.107213 18435 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:47.107843 18435 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:47.111485 18435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c: Generated new TSK 0
I20260619 12:03:47.111725 18435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:03:47.116564 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33271
--builtin_ntp_servers=127.0.203.212:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:03:47.205606 18449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:47.205847 18449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:47.205874 18449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:47.207142 18449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:47.207202 18449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:47.207232 18449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:47.208693 18449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:47.208986 18449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:47.209348 18449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:47.215025 18454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:47.215114 18455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.215205 18449 server_base.cc:1034] running on GCE node
W20260619 12:03:47.215205 18457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.215890 18449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:47.216276 18449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:47.217466 18449 hybrid_clock.cc:648] HybridClock initialized: now 1781870627217440 us; error 42 us; skew 500 ppm
I20260619 12:03:47.219044 18449 webserver.cc:460] Webserver started at http://127.0.203.193:40973/ using document root <none> and password file <none>
I20260619 12:03:47.219343 18449 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:47.219395 18449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:47.219537 18449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:47.220856 18449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "29dc241cfa1e65cf007b6f2f12d6d36e"
server_key_iv: "d01a94a89e5cb9751dd547292fb54e7a"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.221227 18449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "29dc241cfa1e65cf007b6f2f12d6d36e"
server_key_iv: "d01a94a89e5cb9751dd547292fb54e7a"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.222644 18449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:03:47.223469 18463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.223652 18449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:03:47.223762 18449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "29dc241cfa1e65cf007b6f2f12d6d36e"
server_key_iv: "d01a94a89e5cb9751dd547292fb54e7a"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.223842 18449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:47.234830 18449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:47.235203 18449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:47.235323 18449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:47.235584 18449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:47.235971 18449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:47.236013 18449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.236037 18449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:47.236050 18449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.243217 18449 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:40143
I20260619 12:03:47.243312 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:40143 every 8 connection(s)
I20260619 12:03:47.243654 18449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:03:47.250209 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.244631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:56349 (local address 127.0.203.254:33271)
0619 12:03:47.244775 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.244778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.244996 (+ 218us) server_negotiation.cc:407] Connection header received
0619 12:03:47.245854 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.245856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.245872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.245938 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:47.246749 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.247312 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.248419 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.248681 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.248745 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.249770 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:47.249772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:47.249773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:47.249780 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:47.249801 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:47.249807 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:47.249891 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.250071 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.250123 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 12:03:47.250492 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.244573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33271 (local address 127.0.203.193:56349)
0619 12:03:47.244847 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.244888 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.245680 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.246038 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.246042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.246054 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:47.246642 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.246646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.247419 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.247420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.248316 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.248319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.248773 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.248775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.248917 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.249532 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:47.249533 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:47.249572 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:47.249614 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:47.249892 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:47.249899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:47.249903 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.250068 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.250218 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260619 12:03:47.251142 18577 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33271
I20260619 12:03:47.251282 18577 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:47.251509 18577 heartbeater.cc:511] Master 127.0.203.254:33271 requested a full tablet report, sending...
I20260619 12:03:47.252254 18389 ts_manager.cc:194] Registered new tserver with Master: c2fd4caca6de4ff4b0a0b1cc0407d4c0 (127.0.203.193:40143)
I20260619 12:03:47.252753 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18449
I20260619 12:03:47.252851 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:03:47.253049 815 external_mini_cluster.cc:1428] Setting key 03f60e36d0344fe52a51450538fcf944
I20260619 12:03:47.253270 18389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.193:56349
I20260619 12:03:47.254621 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33271
--builtin_ntp_servers=127.0.203.212:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:03:47.340572 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:47.341022 18580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:47.341086 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:47.342486 18580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:47.342566 18580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:47.342600 18580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:47.344604 18580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:47.344985 18580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:47.345405 18580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:47.351610 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:47.351670 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.352002 18580 server_base.cc:1034] running on GCE node
W20260619 12:03:47.351637 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.352334 18580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:47.352741 18580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:47.353967 18580 hybrid_clock.cc:648] HybridClock initialized: now 1781870627353941 us; error 30 us; skew 500 ppm
I20260619 12:03:47.355708 18580 webserver.cc:460] Webserver started at http://127.0.203.194:39875/ using document root <none> and password file <none>
I20260619 12:03:47.356037 18580 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:47.356117 18580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:47.356303 18580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:47.357985 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "184e278b72474b51aed19b92f549e280"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "da72a25c9daab27c3b14304ae7af0faf"
server_key_iv: "df6d8ef94e3762b5fd7de34d765399c1"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.358510 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "184e278b72474b51aed19b92f549e280"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "da72a25c9daab27c3b14304ae7af0faf"
server_key_iv: "df6d8ef94e3762b5fd7de34d765399c1"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.360239 18580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:03:47.361531 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.361740 18580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:47.361840 18580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "184e278b72474b51aed19b92f549e280"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "da72a25c9daab27c3b14304ae7af0faf"
server_key_iv: "df6d8ef94e3762b5fd7de34d765399c1"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.361950 18580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:47.377985 18580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:47.378382 18580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:47.378512 18580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:47.378765 18580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:47.379201 18580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:47.379271 18580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.379319 18580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:47.379340 18580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.387228 18580 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:34859
I20260619 12:03:47.387370 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:34859 every 8 connection(s)
I20260619 12:03:47.387715 18580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:03:47.391093 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18580
I20260619 12:03:47.391212 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:03:47.391410 815 external_mini_cluster.cc:1428] Setting key f0588876b7809856113e1a60cd852585
I20260619 12:03:47.392694 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33271
--builtin_ntp_servers=127.0.203.212:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:03:47.394457 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.388700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:33975 (local address 127.0.203.254:33271)
0619 12:03:47.388807 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.388810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.389179 (+ 369us) server_negotiation.cc:407] Connection header received
0619 12:03:47.389837 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.389839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.389851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.389921 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:47.390684 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.391446 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.392908 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.393131 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.393169 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.394153 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:47.394154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:47.394155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:47.394162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:47.394177 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:47.394182 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:47.394251 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.394327 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.394356 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 12:03:47.394696 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.388579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33271 (local address 127.0.203.194:33975)
0619 12:03:47.388938 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.388984 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.389729 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.389992 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.389996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.390007 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:47.390584 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.390589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.391602 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.391604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.392781 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.392785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.393251 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.393251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.393370 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.393980 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:47.393980 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:47.394013 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:47.394040 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:47.394252 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:47.394259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:47.394262 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.394333 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.394449 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":160,"threads_started":1}
I20260619 12:03:47.395267 18708 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33271
I20260619 12:03:47.395370 18708 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:47.395607 18708 heartbeater.cc:511] Master 127.0.203.254:33271 requested a full tablet report, sending...
I20260619 12:03:47.396035 18388 ts_manager.cc:194] Registered new tserver with Master: 184e278b72474b51aed19b92f549e280 (127.0.203.194:34859)
I20260619 12:03:47.396443 18388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.194:33975
W20260619 12:03:47.485373 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:47.485657 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:47.485692 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:47.486914 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:47.486984 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:47.487026 18711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:47.488425 18711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:47.488772 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:47.489128 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:47.495738 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:47.495757 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.495805 18711 server_base.cc:1034] running on GCE node
W20260619 12:03:47.495757 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:47.496224 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:47.496619 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:47.497888 18711 hybrid_clock.cc:648] HybridClock initialized: now 1781870627497878 us; error 32 us; skew 500 ppm
I20260619 12:03:47.499404 18711 webserver.cc:460] Webserver started at http://127.0.203.195:40767/ using document root <none> and password file <none>
I20260619 12:03:47.499693 18711 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:47.499766 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:47.499953 18711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:47.501233 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "22c78867c2cd454db8ede9d1bd03f85c"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "8fc4a83097bff17f5dbe4a59389e58fd"
server_key_iv: "9adcc858b8dbd658b271f747b87c9804"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.501621 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "22c78867c2cd454db8ede9d1bd03f85c"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "8fc4a83097bff17f5dbe4a59389e58fd"
server_key_iv: "9adcc858b8dbd658b271f747b87c9804"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.503234 18711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 12:03:47.504455 18725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.504827 18711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:47.504963 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "22c78867c2cd454db8ede9d1bd03f85c"
format_stamp: "Formatted at 2026-06-19 12:03:47 on dist-test-slave-fbq8"
server_key: "8fc4a83097bff17f5dbe4a59389e58fd"
server_key_iv: "9adcc858b8dbd658b271f747b87c9804"
server_key_version: "encryptionkey@0"
I20260619 12:03:47.505065 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:47.555805 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:47.556273 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:47.556458 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:47.556872 18711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:47.557300 18711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:47.557360 18711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.557399 18711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:47.557440 18711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:47.564968 18711 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:33125
I20260619 12:03:47.565025 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:33125 every 8 connection(s)
I20260619 12:03:47.565413 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:03:47.570117 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18711
I20260619 12:03:47.570216 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:03:47.570401 815 external_mini_cluster.cc:1428] Setting key a5ee821abd95db557794607312b472d7
I20260619 12:03:47.571879 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.566335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:39259 (local address 127.0.203.254:33271)
0619 12:03:47.566382 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.566384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.566671 (+ 287us) server_negotiation.cc:407] Connection header received
0619 12:03:47.567343 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.567344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.567357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.567403 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:47.568121 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.568654 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.569652 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.569895 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.569972 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.571534 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:47.571536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:47.571537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:47.571543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:47.571563 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:47.571569 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:47.571645 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.571751 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.571790 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 12:03:47.572158 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.566316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:33271 (local address 127.0.203.195:39259)
0619 12:03:47.566518 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.566563 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.567224 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.567475 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.567481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.567498 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:47.568023 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.568027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.568751 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.568752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.569562 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.569564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.570044 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.570046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.570346 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.571339 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:47.571339 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:47.571375 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:47.571405 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:47.571652 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:47.571658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:47.571661 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.571755 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.571895 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260619 12:03:47.572799 18839 heartbeater.cc:347] Connected to a master server at 127.0.203.254:33271
I20260619 12:03:47.572919 18839 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:47.573122 18839 heartbeater.cc:511] Master 127.0.203.254:33271 requested a full tablet report, sending...
I20260619 12:03:47.573534 18388 ts_manager.cc:194] Registered new tserver with Master: 22c78867c2cd454db8ede9d1bd03f85c (127.0.203.195:33125)
I20260619 12:03:47.573885 18388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.195:39259
I20260619 12:03:47.582540 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:03:47.587628 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.584143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.0.203.254:33271)
0619 12:03:47.584241 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.584243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.584417 (+ 174us) server_negotiation.cc:407] Connection header received
0619 12:03:47.584478 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.584479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.584490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.584548 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:47.584757 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.585435 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.586293 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.586482 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.586549 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.587335 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:47.587336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:47.587336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:47.587341 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:47.587358 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:47.587362 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:47.587415 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.587503 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.587526 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:03:47.589785 18388 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:03:47.591745 18388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:03:47.600095 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.594586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:34859 (local address 127.0.0.1:33328)
0619 12:03:47.595773 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.595783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.595838 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.596081 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.596083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.596085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:47.596133 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.596134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.597417 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.597418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.598617 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.598620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.599714 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.599715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.599806 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.599829 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.599919 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.599995 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":82,"threads_started":1}
I20260619 12:03:47.600081 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.594779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.0.203.194:34859)
0619 12:03:47.595067 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.595073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.595901 (+ 828us) server_negotiation.cc:407] Connection header received
0619 12:03:47.595954 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.595956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.595972 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.596053 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:47.596234 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.597344 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.598758 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.599495 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.599572 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.599678 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.599934 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.599979 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":148,"threads_started":1}
I20260619 12:03:47.600174 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.594869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.0.203.193:40143)
0619 12:03:47.595074 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.595078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.595518 (+ 440us) server_negotiation.cc:407] Connection header received
0619 12:03:47.595568 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.595571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.595585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.595676 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:47.595891 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.597142 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.598660 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.599545 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.599609 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.599708 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.600043 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.600086 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
I20260619 12:03:47.600404 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.594576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40143 (local address 127.0.0.1:57434)
0619 12:03:47.595110 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.595160 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.595252 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.595677 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.595680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.595687 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:47.595768 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.595771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.597242 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.597243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.598573 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.598576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.599744 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.599745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.599804 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.599828 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.599898 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.599964 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":99,"threads_started":1}
I20260619 12:03:47.600867 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.594576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:33125 (local address 127.0.0.1:35192)
0619 12:03:47.595377 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.595388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.595448 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.595676 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.595680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.595683 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:47.595789 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.595792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.596900 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.596901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.599499 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.599502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.600584 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.600585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.600673 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.600678 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.600727 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.600792 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":1302,"thread_start_us":85,"threads_started":1}
I20260619 12:03:47.601215 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.594909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.203.195:33125)
0619 12:03:47.595230 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.595237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.595483 (+ 246us) server_negotiation.cc:407] Connection header received
0619 12:03:47.595532 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.595534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.595549 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.595646 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:47.595875 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.596813 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.599614 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.600379 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.600451 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.600543 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.601077 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.601126 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
I20260619 12:03:47.601234 18511 tablet_service.cc:1467] Processing CreateTablet for tablet 2f990e4708624ed5a96404d9522d1474 (DEFAULT_TABLE table=test-table [id=18801c4337de4f3b8d8a57e078ae1d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:47.601235 18642 tablet_service.cc:1467] Processing CreateTablet for tablet 2f990e4708624ed5a96404d9522d1474 (DEFAULT_TABLE table=test-table [id=18801c4337de4f3b8d8a57e078ae1d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:47.601629 18511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f990e4708624ed5a96404d9522d1474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:47.601629 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f990e4708624ed5a96404d9522d1474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:47.602160 18773 tablet_service.cc:1467] Processing CreateTablet for tablet 2f990e4708624ed5a96404d9522d1474 (DEFAULT_TABLE table=test-table [id=18801c4337de4f3b8d8a57e078ae1d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:47.602583 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f990e4708624ed5a96404d9522d1474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:47.605257 18854 tablet_bootstrap.cc:492] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Bootstrap starting.
I20260619 12:03:47.605631 18855 tablet_bootstrap.cc:492] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Bootstrap starting.
I20260619 12:03:47.606105 18854 tablet_bootstrap.cc:654] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:47.606388 18855 tablet_bootstrap.cc:654] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:47.606495 18854 log.cc:824] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:47.606739 18855 log.cc:824] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:47.606922 18853 tablet_bootstrap.cc:492] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: Bootstrap starting.
I20260619 12:03:47.607385 18854 tablet_bootstrap.cc:492] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: No bootstrap required, opened a new log
I20260619 12:03:47.607467 18854 ts_tablet_manager.cc:1397] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:03:47.607641 18855 tablet_bootstrap.cc:492] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: No bootstrap required, opened a new log
I20260619 12:03:47.607672 18853 tablet_bootstrap.cc:654] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:47.607728 18855 ts_tablet_manager.cc:1397] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:03:47.608000 18853 log.cc:824] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:47.608994 18853 tablet_bootstrap.cc:492] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: No bootstrap required, opened a new log
I20260619 12:03:47.609079 18853 ts_tablet_manager.cc:1397] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:03:47.610255 18855 raft_consensus.cc:348] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.610432 18855 raft_consensus.cc:374] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:47.610476 18855 raft_consensus.cc:729] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c78867c2cd454db8ede9d1bd03f85c, State: Initialized, Role: FOLLOWER
I20260619 12:03:47.610589 18855 consensus_queue.cc:260] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.610661 18854 raft_consensus.cc:348] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.610845 18854 raft_consensus.cc:374] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:47.610877 18854 raft_consensus.cc:729] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184e278b72474b51aed19b92f549e280, State: Initialized, Role: FOLLOWER
I20260619 12:03:47.610879 18855 ts_tablet_manager.cc:1428] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:47.611023 18839 heartbeater.cc:503] Master 127.0.203.254:33271 was elected leader, sending a full tablet report...
I20260619 12:03:47.611025 18854 consensus_queue.cc:260] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.611310 18854 ts_tablet_manager.cc:1428] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 12:03:47.611482 18708 heartbeater.cc:503] Master 127.0.203.254:33271 was elected leader, sending a full tablet report...
I20260619 12:03:47.611907 18853 raft_consensus.cc:348] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.612094 18853 raft_consensus.cc:374] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:47.612155 18853 raft_consensus.cc:729] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2fd4caca6de4ff4b0a0b1cc0407d4c0, State: Initialized, Role: FOLLOWER
I20260619 12:03:47.612300 18853 consensus_queue.cc:260] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.612562 18853 ts_tablet_manager.cc:1428] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 12:03:47.612692 18577 heartbeater.cc:503] Master 127.0.203.254:33271 was elected leader, sending a full tablet report...
W20260619 12:03:47.638392 18709 tablet.cc:2369] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:03:47.744536 18578 tablet.cc:2369] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:03:47.816083 18840 tablet.cc:2369] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:03:47.960486 18860 raft_consensus.cc:482] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:47.960717 18860 raft_consensus.cc:504] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.961234 18860 leader_election.cc:288] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22c78867c2cd454db8ede9d1bd03f85c (127.0.203.195:33125), c2fd4caca6de4ff4b0a0b1cc0407d4c0 (127.0.203.193:40143)
I20260619 12:03:47.964742 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.961506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40143 (local address 127.0.203.194:40023)
0619 12:03:47.961738 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.961758 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.961913 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.962153 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.962154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.962156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:47.962279 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.962282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.962965 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.962966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.963834 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.963836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.964391 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.964392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.964481 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.964487 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.964546 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.964594 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":27,"thread_start_us":98,"threads_started":1}
I20260619 12:03:47.964754 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.961609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:40023 (local address 127.0.203.193:40143)
0619 12:03:47.961693 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.961696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.961946 (+ 250us) server_negotiation.cc:407] Connection header received
0619 12:03:47.962032 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.962033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.962047 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.962140 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:47.962362 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.962840 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.963930 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.964300 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.964361 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.964368 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.964585 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.964628 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:03:47.964838 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.961636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:47281 (local address 127.0.203.195:33125)
0619 12:03:47.961695 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:47.961698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:47.961825 (+ 127us) server_negotiation.cc:407] Connection header received
0619 12:03:47.961901 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:47.961903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:47.961914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:47.961988 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:47.962227 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.962823 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.963996 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.964415 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.964494 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.964504 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:47.964714 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 12:03:47.964769 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 12:03:47.964860 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:47.961505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:33125 (local address 127.0.203.194:47281)
0619 12:03:47.961722 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:47.961734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:47.961810 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:47.962042 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:47.962044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:47.962048 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:47.962131 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.962134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.962921 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.962922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.963864 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:47.963867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:47.964512 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:47.964513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:47.964614 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:47.964620 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:03:47.964707 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:03:47.964756 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":22,"thread_start_us":132,"threads_started":1}
I20260619 12:03:47.965440 18531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f990e4708624ed5a96404d9522d1474" candidate_uuid: "184e278b72474b51aed19b92f549e280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" is_pre_election: true
I20260619 12:03:47.965548 18793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f990e4708624ed5a96404d9522d1474" candidate_uuid: "184e278b72474b51aed19b92f549e280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c78867c2cd454db8ede9d1bd03f85c" is_pre_election: true
I20260619 12:03:47.965647 18531 raft_consensus.cc:2454] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184e278b72474b51aed19b92f549e280 in term 0.
I20260619 12:03:47.965674 18793 raft_consensus.cc:2454] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184e278b72474b51aed19b92f549e280 in term 0.
I20260619 12:03:47.965935 18597 leader_election.cc:302] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184e278b72474b51aed19b92f549e280, c2fd4caca6de4ff4b0a0b1cc0407d4c0; no voters:
I20260619 12:03:47.966099 18860 raft_consensus.cc:2781] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:03:47.966157 18860 raft_consensus.cc:482] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:03:47.966208 18860 raft_consensus.cc:3037] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:47.967108 18860 raft_consensus.cc:504] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.967284 18860 leader_election.cc:288] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [CANDIDATE]: Term 1 election: Requested vote from peers 22c78867c2cd454db8ede9d1bd03f85c (127.0.203.195:33125), c2fd4caca6de4ff4b0a0b1cc0407d4c0 (127.0.203.193:40143)
I20260619 12:03:47.967453 18793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f990e4708624ed5a96404d9522d1474" candidate_uuid: "184e278b72474b51aed19b92f549e280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c78867c2cd454db8ede9d1bd03f85c"
I20260619 12:03:47.967496 18531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f990e4708624ed5a96404d9522d1474" candidate_uuid: "184e278b72474b51aed19b92f549e280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0"
I20260619 12:03:47.967589 18793 raft_consensus.cc:3037] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:47.967628 18531 raft_consensus.cc:3037] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:47.968458 18793 raft_consensus.cc:2454] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184e278b72474b51aed19b92f549e280 in term 1.
I20260619 12:03:47.968506 18531 raft_consensus.cc:2454] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184e278b72474b51aed19b92f549e280 in term 1.
I20260619 12:03:47.968717 18595 leader_election.cc:302] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184e278b72474b51aed19b92f549e280, 22c78867c2cd454db8ede9d1bd03f85c; no voters:
I20260619 12:03:47.968830 18860 raft_consensus.cc:2781] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:47.969008 18860 raft_consensus.cc:686] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 1 LEADER]: Becoming Leader. State: Replica: 184e278b72474b51aed19b92f549e280, State: Running, Role: LEADER
I20260619 12:03:47.969094 18860 consensus_queue.cc:237] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } }
I20260619 12:03:47.970088 18388 catalog_manager.cc:5495] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184e278b72474b51aed19b92f549e280 (127.0.203.194). New cstate: current_term: 1 leader_uuid: "184e278b72474b51aed19b92f549e280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184e278b72474b51aed19b92f549e280" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 34859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 } health_report { overall_health: UNKNOWN } } }
I20260619 12:03:48.045956 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:48.042890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.203.194:34859)
0619 12:03:48.043138 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:48.043140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:48.043151 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:48.043190 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:48.043190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:48.043217 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:48.043298 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:48.043528 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:48.044028 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:48.045087 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:48.045295 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:48.045368 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:48.045554 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:48.045671 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:03:48.045742 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:48.045810 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 12:03:48.045830 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260619 12:03:48.048324 18793 raft_consensus.cc:1261] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 1 FOLLOWER]: Refusing update from remote peer 184e278b72474b51aed19b92f549e280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:48.048377 18531 raft_consensus.cc:1261] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 1 FOLLOWER]: Refusing update from remote peer 184e278b72474b51aed19b92f549e280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:48.048812 18864 consensus_queue.cc:1035] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c78867c2cd454db8ede9d1bd03f85c" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:48.049026 18860 consensus_queue.cc:1035] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2fd4caca6de4ff4b0a0b1cc0407d4c0" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 40143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:48.052160 18868 mvcc.cc:205] Tried to move back new op lower bound from 7298542092482723840 to 7298542092162318336. Current Snapshot: MvccSnapshot[applied={T|T < 7298542092482723840}]
I20260619 12:03:48.052335 18867 mvcc.cc:205] Tried to move back new op lower bound from 7298542092482723840 to 7298542092162318336. Current Snapshot: MvccSnapshot[applied={T|T < 7298542092482723840 or (T in {7298542092482723840})}]
I20260619 12:03:48.054432 18389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:48.054571 18389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:48.055559 18389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1b3f78199c04272b497a7cbfbfa7d2c: Sending DeleteTablet for 3 replicas of tablet 2f990e4708624ed5a96404d9522d1474
I20260619 12:03:48.055939 18642 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f990e4708624ed5a96404d9522d1474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:48 UTC) from {username='slave'} at 127.0.0.1:33328
I20260619 12:03:48.055943 18511 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f990e4708624ed5a96404d9522d1474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:48 UTC) from {username='slave'} at 127.0.0.1:57434
I20260619 12:03:48.056210 18773 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f990e4708624ed5a96404d9522d1474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:48 UTC) from {username='slave'} at 127.0.0.1:35192
I20260619 12:03:48.056288 18877 tablet_replica.cc:330] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0: stopping tablet replica
I20260619 12:03:48.056391 18877 raft_consensus.cc:2229] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:48.056455 18876 tablet_replica.cc:330] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: stopping tablet replica
I20260619 12:03:48.056506 18877 raft_consensus.cc:2258] T 2f990e4708624ed5a96404d9522d1474 P c2fd4caca6de4ff4b0a0b1cc0407d4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:48.056555 18876 raft_consensus.cc:2229] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:03:48.056666 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18449
I20260619 12:03:48.058848 18876 raft_consensus.cc:2258] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:48.059310 18876 ts_tablet_manager.cc:1905] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:48.060787 18878 tablet_replica.cc:330] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: stopping tablet replica
I20260619 12:03:48.060920 18878 raft_consensus.cc:2229] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:48.060999 18876 ts_tablet_manager.cc:1918] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:48.061091 18876 log.cc:1196] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/2f990e4708624ed5a96404d9522d1474
I20260619 12:03:48.061160 18878 raft_consensus.cc:2258] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:48.061386 18876 ts_tablet_manager.cc:1939] T 2f990e4708624ed5a96404d9522d1474 P 184e278b72474b51aed19b92f549e280: Deleting consensus metadata
I20260619 12:03:48.061697 18878 ts_tablet_manager.cc:1905] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:48.061849 18376 catalog_manager.cc:4841] TS 184e278b72474b51aed19b92f549e280 (127.0.203.194:34859): tablet 2f990e4708624ed5a96404d9522d1474 (table test-table [id=18801c4337de4f3b8d8a57e078ae1d6c]) successfully deleted
I20260619 12:03:48.063053 18878 ts_tablet_manager.cc:1918] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:48.063138 18878 log.cc:1196] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/2f990e4708624ed5a96404d9522d1474
W20260619 12:03:48.063458 18375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:40143 (error 108)
I20260619 12:03:48.063733 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18580
I20260619 12:03:48.063946 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:48.063791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:40143 (local address 127.0.0.1:57448)
0619 12:03:48.063871 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:48.063907 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 12:03:48.064819 18375 catalog_manager.cc:4584] TS c2fd4caca6de4ff4b0a0b1cc0407d4c0 (127.0.203.193:40143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f990e4708624ed5a96404d9522d1474: Network error: Client connection negotiation failed: client connection to 127.0.203.193:40143: connect: Connection refused (error 111)
I20260619 12:03:48.065083 18878 ts_tablet_manager.cc:1939] T 2f990e4708624ed5a96404d9522d1474 P 22c78867c2cd454db8ede9d1bd03f85c: Deleting consensus metadata
I20260619 12:03:48.065506 18373 catalog_manager.cc:4841] TS 22c78867c2cd454db8ede9d1bd03f85c (127.0.203.195:33125): tablet 2f990e4708624ed5a96404d9522d1474 (table test-table [id=18801c4337de4f3b8d8a57e078ae1d6c]) successfully deleted
I20260619 12:03:48.069623 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18711
I20260619 12:03:48.075358 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18358
2026-06-19T12:03:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1167 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:48 dist-test-slave-fbq8 krb5kdc[18882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:48 dist-test-slave-fbq8 krb5kdc[18882](info): set up 2 sockets
Jun 19 12:03:48 dist-test-slave-fbq8 krb5kdc[18882](info): commencing operation
krb5kdc: starting...
W20260619 12:03:49.164939 815 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:03:49.300845 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:35459
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:35459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:49.390234 18895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:49.390458 18895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:49.390481 18895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:49.391637 18895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:49.391690 18895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:49.391705 18895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:49.391717 18895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:49.391731 18895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:49.393251 18895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:35459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:49.393527 18895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:49.393807 18895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:49.399648 18900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:49.399761 18895 server_base.cc:1034] running on GCE node
W20260619 12:03:49.399716 18903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:49.399736 18901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:49.400317 18895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:49.400614 18895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:49.401859 18895 hybrid_clock.cc:648] HybridClock initialized: now 1781870629401853 us; error 37 us; skew 500 ppm
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:49.404950 18895 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:03:49.405558 18895 webserver.cc:460] Webserver started at http://127.0.203.254:35031/ using document root <none> and password file <none>
I20260619 12:03:49.405822 18895 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:49.405896 18895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:49.406056 18895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:49.407274 18895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "36caab4a3c9e4eac8c8010ff12091d0d"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "3617010ebb78fccc6b18283a7f4b1fae"
server_key_iv: "88beb01343355c909bb4e36c1425ff68"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.407660 18895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "36caab4a3c9e4eac8c8010ff12091d0d"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "3617010ebb78fccc6b18283a7f4b1fae"
server_key_iv: "88beb01343355c909bb4e36c1425ff68"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.409418 18895 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 12:03:49.410498 18910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:49.410868 18895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:49.411010 18895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "36caab4a3c9e4eac8c8010ff12091d0d"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "3617010ebb78fccc6b18283a7f4b1fae"
server_key_iv: "88beb01343355c909bb4e36c1425ff68"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.411103 18895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:49.434818 18895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:49.437283 18895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:49.437454 18895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:49.442468 18895 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:35459
I20260619 12:03:49.442543 18962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:35459 every 8 connection(s)
I20260619 12:03:49.442940 18895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:49.443970 18963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:49.446834 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18895
I20260619 12:03:49.447026 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:49.447227 815 external_mini_cluster.cc:1428] Setting key 1c3d2b249152d6e64132021055613584
I20260619 12:03:49.447763 18963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d: Bootstrap starting.
I20260619 12:03:49.448621 18963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:49.449054 18963 log.cc:824] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:49.450013 18963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d: No bootstrap required, opened a new log
I20260619 12:03:49.452523 18963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35459 } }
I20260619 12:03:49.452760 18963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:49.452792 18963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36caab4a3c9e4eac8c8010ff12091d0d, State: Initialized, Role: FOLLOWER
I20260619 12:03:49.452967 18963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35459 } }
I20260619 12:03:49.453056 18963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:49.453083 18963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:49.453166 18963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:49.454018 18963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35459 } }
I20260619 12:03:49.454151 18963 leader_election.cc:302] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36caab4a3c9e4eac8c8010ff12091d0d; no voters:
I20260619 12:03:49.454383 18963 leader_election.cc:288] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:49.454409 18968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:49.454545 18968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [term 1 LEADER]: Becoming Leader. State: Replica: 36caab4a3c9e4eac8c8010ff12091d0d, State: Running, Role: LEADER
I20260619 12:03:49.454727 18968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35459 } }
I20260619 12:03:49.454727 18963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:49.457084 18969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36caab4a3c9e4eac8c8010ff12091d0d. Latest consensus state: current_term: 1 leader_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35459 } } }
W20260619 12:03:49.457156 18983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 12:03:49.457257 18969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [sys.catalog]: This master's current role is: LEADER
W20260619 12:03:49.457262 18983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:03:49.457085 18970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36caab4a3c9e4eac8c8010ff12091d0d" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 35459 } } }
I20260619 12:03:49.457338 18970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:49.457500 18985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:49.457692 18985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:49.459666 18985 catalog_manager.cc:1347] Generated new cluster ID: 68a9c5a892f14021ade7b290e032884a
I20260619 12:03:49.459733 18985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:49.460747 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:49.447990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.0.203.254:35459)
0619 12:03:49.448185 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:49.448194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:49.448231 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:03:49.448968 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:49.448977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:49.449033 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:49.449225 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:49.450171 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.451168 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.452413 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.453099 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.453241 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:49.456384 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:49.456389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:49.456389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:49.456407 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:49.458721 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:49.459203 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:49.459207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:49.459207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:49.459329 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:49.459547 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:49.459548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:49.459548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:49.459873 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:49.460033 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:49.460296 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 12:03:49.460464 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260619 12:03:49.482232 18985 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:49.483122 18985 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:49.490684 18985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d: Generated new TSK 0
I20260619 12:03:49.490897 18985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:03:49.523571 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35459
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:49.614009 18989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:49.614300 18989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:49.614347 18989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:49.615677 18989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:49.615751 18989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:49.615797 18989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:49.617477 18989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:49.617784 18989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:49.618037 18989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:49.623620 18995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:49.623781 18989 server_base.cc:1034] running on GCE node
W20260619 12:03:49.623585 18994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:49.623589 18997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:49.624272 18989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:49.624557 18989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:49.625746 18989 hybrid_clock.cc:648] HybridClock initialized: now 1781870629625714 us; error 53 us; skew 500 ppm
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:49.628901 18989 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:03:49.629501 18989 webserver.cc:460] Webserver started at http://127.0.203.193:42169/ using document root <none> and password file <none>
I20260619 12:03:49.629791 18989 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:49.629849 18989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:49.630044 18989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:49.631338 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "28878ebbc4a940219e1766c7f700f4a1"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "12fbdc839583d0d3a7e62f00366ed643"
server_key_iv: "be1932792bb1e34b8e93291ee746150f"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.631743 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "28878ebbc4a940219e1766c7f700f4a1"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "12fbdc839583d0d3a7e62f00366ed643"
server_key_iv: "be1932792bb1e34b8e93291ee746150f"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.633481 18989 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:03:49.634570 19004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:49.634917 18989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:49.635043 18989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "28878ebbc4a940219e1766c7f700f4a1"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "12fbdc839583d0d3a7e62f00366ed643"
server_key_iv: "be1932792bb1e34b8e93291ee746150f"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.635146 18989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:49.649716 18989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:49.652064 18989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:49.652256 18989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:49.652525 18989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:49.652907 18989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:49.652962 18989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:49.653005 18989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:49.653054 18989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:49.660614 18989 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:45409
I20260619 12:03:49.660688 19117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:45409 every 8 connection(s)
I20260619 12:03:49.661031 18989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:49.670517 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 18989
I20260619 12:03:49.670661 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:03:49.670909 815 external_mini_cluster.cc:1428] Setting key 38d1f6a9bfa9faf98dcc052a1c44fc69
I20260619 12:03:49.671693 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:49.661953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:56737 (local address 127.0.203.254:35459)
0619 12:03:49.662062 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:49.662069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:49.662237 (+ 168us) server_negotiation.cc:407] Connection header received
0619 12:03:49.663160 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:49.663162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:49.663174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:49.663251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:49.663896 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.664536 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.665617 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.665851 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.665918 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:49.668299 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:49.668300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:49.668301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:49.668308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:49.669846 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:49.670340 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:49.670341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:49.670342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:49.670409 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:49.670779 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:49.670779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:49.670780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:49.670932 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:49.671016 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:49.671443 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 12:03:49.671578 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 12:03:49.671825 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:49.661909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35459 (local address 127.0.203.193:56737)
0619 12:03:49.662085 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:49.662129 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:49.663041 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:49.663315 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:49.663318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:49.663577 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:49.663773 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:49.663777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.664628 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.664629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:49.665521 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:49.665524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.665937 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.665938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:49.666075 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:49.666576 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:49.666584 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:49.668199 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:49.669991 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:49.669998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:49.669999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:49.670240 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:49.670495 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:49.670496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:49.670496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:49.670615 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:49.671061 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:49.671069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:49.671249 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 12:03:49.671391 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 12:03:49.671568 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260619 12:03:49.672523 19118 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35459
I20260619 12:03:49.672706 19118 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:49.672964 19118 heartbeater.cc:511] Master 127.0.203.254:35459 requested a full tablet report, sending...
I20260619 12:03:49.673648 18927 ts_manager.cc:194] Registered new tserver with Master: 28878ebbc4a940219e1766c7f700f4a1 (127.0.203.193:45409)
I20260619 12:03:49.674384 18927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:56737
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:03:49.700956 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35459
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:49.791335 19123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:49.791574 19123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:49.791597 19123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:49.792811 19123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:49.792857 19123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:49.792884 19123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:49.794380 19123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:49.794634 19123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:49.794899 19123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:49.801548 19128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:49.801553 19131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:49.801553 19129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:49.801893 19123 server_base.cc:1034] running on GCE node
I20260619 12:03:49.802287 19123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:49.802572 19123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:49.803818 19123 hybrid_clock.cc:648] HybridClock initialized: now 1781870629803776 us; error 58 us; skew 500 ppm
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:49.807302 19123 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:03:49.807883 19123 webserver.cc:460] Webserver started at http://127.0.203.194:38265/ using document root <none> and password file <none>
I20260619 12:03:49.808144 19123 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:49.808218 19123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:49.808455 19123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:49.809734 19123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "1986c5c8604b4a1fa8e2021ac6235ed7"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "d9ddc123e32125854390bbf8bef88600"
server_key_iv: "8e1fc4aa4fc6926140330f4efd801b7f"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.810153 19123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "1986c5c8604b4a1fa8e2021ac6235ed7"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "d9ddc123e32125854390bbf8bef88600"
server_key_iv: "8e1fc4aa4fc6926140330f4efd801b7f"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.811767 19123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 12:03:49.812783 19138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:49.813081 19123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 12:03:49.813205 19123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "1986c5c8604b4a1fa8e2021ac6235ed7"
format_stamp: "Formatted at 2026-06-19 12:03:49 on dist-test-slave-fbq8"
server_key: "d9ddc123e32125854390bbf8bef88600"
server_key_iv: "8e1fc4aa4fc6926140330f4efd801b7f"
server_key_version: "encryptionkey@0"
I20260619 12:03:49.813308 19123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:49.828639 19123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:49.830896 19123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:49.831076 19123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:49.831333 19123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:49.831673 19123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:49.831729 19123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:49.831773 19123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:49.831802 19123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:49.838804 19123 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:40477
I20260619 12:03:49.838922 19251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:40477 every 8 connection(s)
I20260619 12:03:49.839325 19123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:49.847900 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 19123
I20260619 12:03:49.848140 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:03:49.848381 815 external_mini_cluster.cc:1428] Setting key f3f7eb09c90b0faf69ba91d294d2ac2a
I20260619 12:03:49.850849 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:49.840381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:42733 (local address 127.0.203.254:35459)
0619 12:03:49.840517 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:49.840520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:49.840611 (+ 91us) server_negotiation.cc:407] Connection header received
0619 12:03:49.841768 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:49.841770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:49.841781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:49.841849 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:49.842578 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.843090 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.844245 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.844433 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.844502 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:49.847426 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:49.847428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:49.847428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:49.847437 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:49.849157 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:49.849708 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:49.849709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:49.849709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:49.849772 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:49.850089 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:49.850089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:49.850090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:49.850209 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:49.850270 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:49.850701 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 12:03:49.850776 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 12:03:49.851244 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:49.840264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35459 (local address 127.0.203.194:42733)
0619 12:03:49.840457 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:49.840516 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:49.841636 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:49.841938 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:49.841942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:49.842257 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:49.842490 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:49.842494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.843200 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.843201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:49.844145 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:49.844149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:49.844548 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:49.844549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:49.844726 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:49.845548 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:49.845566 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:49.847245 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:49.849249 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:49.849256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:49.849259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:49.849617 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:49.849855 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:49.849856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:49.849856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:49.850013 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:49.850282 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:49.850287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:49.850476 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 12:03:49.850604 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 12:03:49.850821 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260619 12:03:49.851900 19252 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35459
I20260619 12:03:49.852008 19252 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:49.852234 19252 heartbeater.cc:511] Master 127.0.203.254:35459 requested a full tablet report, sending...
I20260619 12:03:49.852813 18927 ts_manager.cc:194] Registered new tserver with Master: 1986c5c8604b4a1fa8e2021ac6235ed7 (127.0.203.194:40477)
I20260619 12:03:49.853232 18927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:42733
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:03:49.880828 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35459
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:36981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:49.974882 19257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:49.975116 19257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:49.975142 19257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:49.976821 19257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:49.976915 19257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:49.976965 19257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:49.979363 19257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:36981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:49.979806 19257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:49.980105 19257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:49.992558 19265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:49.992810 19257 server_base.cc:1034] running on GCE node
W20260619 12:03:49.992558 19263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:49.992748 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:49.993292 19257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:49.993655 19257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:49.994850 19257 hybrid_clock.cc:648] HybridClock initialized: now 1781870629994834 us; error 41 us; skew 500 ppm
Jun 19 12:03:49 dist-test-slave-fbq8 krb5kdc[18882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:49.998688 19257 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:03:49.999681 19257 webserver.cc:460] Webserver started at http://127.0.203.195:45365/ using document root <none> and password file <none>
I20260619 12:03:49.999979 19257 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:50.000058 19257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:50.000808 19257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:50.002298 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "6290ec310452409390b36f0c06b00ec9"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "bcde43fe4f70e57386645aeecde3d118"
server_key_iv: "531043d694ff6f1752e212095e084033"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.002696 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "6290ec310452409390b36f0c06b00ec9"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "bcde43fe4f70e57386645aeecde3d118"
server_key_iv: "531043d694ff6f1752e212095e084033"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.004460 19257 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260619 12:03:50.005740 19272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.006134 19257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:50.006268 19257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "6290ec310452409390b36f0c06b00ec9"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "bcde43fe4f70e57386645aeecde3d118"
server_key_iv: "531043d694ff6f1752e212095e084033"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.006361 19257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:50.032356 19257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:50.035064 19257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:50.035269 19257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:50.035545 19257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:50.035892 19257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:50.035959 19257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.035997 19257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:50.036026 19257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.043258 19257 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:46125
I20260619 12:03:50.043308 19385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:46125 every 8 connection(s)
I20260619 12:03:50.043665 19257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:03:50.048520 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 19257
I20260619 12:03:50.048686 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:03:50.048880 815 external_mini_cluster.cc:1428] Setting key 96f469d4655acf59ac4e70c4e7c9fb32
Jun 19 12:03:50 dist-test-slave-fbq8 krb5kdc[18882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:50.056169 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.044773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:52063 (local address 127.0.203.254:35459)
0619 12:03:50.044842 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.044844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.045042 (+ 198us) server_negotiation.cc:407] Connection header received
0619 12:03:50.046415 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.046417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.046429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.046505 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:50.047371 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.048067 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.049799 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.050034 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.050133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.052837 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:50.052841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:50.052841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:50.052858 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:50.054556 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:50.055032 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:50.055033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:50.055034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:50.055098 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:50.055373 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:50.055374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:50.055374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:50.055577 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:50.055652 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.055970 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.056060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:03:50.056409 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.044696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:35459 (local address 127.0.203.195:52063)
0619 12:03:50.044895 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.044940 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.046155 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.046609 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.046614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.046981 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:50.047240 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.047246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.048353 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.048354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.049661 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.049666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.050142 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.050143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.050299 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.050932 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:50.050942 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:50.052690 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:50.054677 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:50.054685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:50.054688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:50.054941 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:50.055169 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:50.055170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:50.055170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:50.055295 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:50.055654 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:50.055662 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:50.055822 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.055938 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.056122 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":122,"threads_started":1}
I20260619 12:03:50.057096 19386 heartbeater.cc:347] Connected to a master server at 127.0.203.254:35459
I20260619 12:03:50.057206 19386 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:50.057438 19386 heartbeater.cc:511] Master 127.0.203.254:35459 requested a full tablet report, sending...
I20260619 12:03:50.057993 18927 ts_manager.cc:194] Registered new tserver with Master: 6290ec310452409390b36f0c06b00ec9 (127.0.203.195:46125)
I20260619 12:03:50.058418 18927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:52063
I20260619 12:03:50.061239 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:03:50.070122 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.062719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.0.203.254:35459)
0619 12:03:50.062794 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.062796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.062987 (+ 191us) server_negotiation.cc:407] Connection header received
0619 12:03:50.063044 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.063045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.063055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.063113 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:50.063734 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.064294 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.065317 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.065522 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.065585 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.067182 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:50.067184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:50.067184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:50.067191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:50.068694 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:50.069187 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:50.069187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:50.069188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:50.069241 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:50.069532 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:50.069533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:50.069533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:50.069658 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:50.069748 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.069933 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.069990 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:03:50.072209 18927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:03:50.074208 18927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:03:50.082248 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.077596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:46125 (local address 127.0.0.1:39288)
0619 12:03:50.077731 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.077745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.077808 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.078487 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.078489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.078492 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.078562 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.078564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.080130 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.080131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.081160 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.081164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.081914 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.081914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.081996 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.082011 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.082082 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.082156 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260619 12:03:50.082340 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.077348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:40477 (local address 127.0.0.1:41980)
0619 12:03:50.077631 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.077688 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.077764 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.078602 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.078603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.078605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.078657 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.078658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.079596 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.079597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.081086 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.081089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.082056 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.082057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.082117 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.082120 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.082183 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.082225 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":68,"thread_start_us":84,"threads_started":1}
I20260619 12:03:50.082666 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.078033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.0.203.195:46125)
0619 12:03:50.078274 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.078279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.078290 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:50.078332 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.078334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.078352 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.078452 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.078662 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.079857 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.081258 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.081804 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.081912 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.081983 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.082535 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.082584 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
I20260619 12:03:50.082746 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.077489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.203.194:40477)
0619 12:03:50.078299 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.078305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.078317 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:03:50.078369 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.078376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.078392 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.078495 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.078721 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.079464 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.081190 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.081973 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.082060 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.082128 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.082199 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.082246 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":111,"threads_started":1}
I20260619 12:03:50.083204 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.077795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.203.193:45409)
0619 12:03:50.078019 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.078025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.078036 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:50.078085 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.078087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.078100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.078218 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.078518 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.079823 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.081339 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.082115 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.082186 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.082235 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.083073 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.083127 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260619 12:03:50.083381 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.077450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:45409 (local address 127.0.0.1:60590)
0619 12:03:50.077632 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.077688 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.077764 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.078331 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.078335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.078338 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.078430 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.078433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.079972 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.079973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.081244 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.081246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.082278 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.082280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.082569 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.082575 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.082627 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.082690 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":48,"thread_start_us":114,"threads_started":1}
I20260619 12:03:50.083549 19320 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb182e8b73b4b83859df3b675f5d52d (DEFAULT_TABLE table=test-table [id=dbed73b0a73c4b6d887a74d2eb94c2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:50.083865 19186 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb182e8b73b4b83859df3b675f5d52d (DEFAULT_TABLE table=test-table [id=dbed73b0a73c4b6d887a74d2eb94c2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:50.083948 19320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb182e8b73b4b83859df3b675f5d52d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:50.084153 19052 tablet_service.cc:1467] Processing CreateTablet for tablet 2eb182e8b73b4b83859df3b675f5d52d (DEFAULT_TABLE table=test-table [id=dbed73b0a73c4b6d887a74d2eb94c2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:50.084280 19186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb182e8b73b4b83859df3b675f5d52d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:50.084475 19052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eb182e8b73b4b83859df3b675f5d52d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:50.087954 19403 tablet_bootstrap.cc:492] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Bootstrap starting.
I20260619 12:03:50.088590 19405 tablet_bootstrap.cc:492] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Bootstrap starting.
I20260619 12:03:50.089133 19403 tablet_bootstrap.cc:654] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:50.089705 19403 log.cc:824] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:50.089697 19404 tablet_bootstrap.cc:492] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1: Bootstrap starting.
I20260619 12:03:50.089775 19405 tablet_bootstrap.cc:654] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:50.090551 19403 tablet_bootstrap.cc:492] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: No bootstrap required, opened a new log
I20260619 12:03:50.090647 19403 ts_tablet_manager.cc:1397] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:03:50.090741 19404 tablet_bootstrap.cc:654] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:50.090757 19405 log.cc:824] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:50.091470 19404 log.cc:824] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:50.091773 19405 tablet_bootstrap.cc:492] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: No bootstrap required, opened a new log
I20260619 12:03:50.091881 19405 ts_tablet_manager.cc:1397] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:03:50.092350 19404 tablet_bootstrap.cc:492] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1: No bootstrap required, opened a new log
I20260619 12:03:50.092490 19404 ts_tablet_manager.cc:1397] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:50.092958 19403 raft_consensus.cc:348] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.093168 19403 raft_consensus.cc:374] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:50.093218 19403 raft_consensus.cc:729] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6290ec310452409390b36f0c06b00ec9, State: Initialized, Role: FOLLOWER
I20260619 12:03:50.093380 19403 consensus_queue.cc:260] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.093746 19386 heartbeater.cc:503] Master 127.0.203.254:35459 was elected leader, sending a full tablet report...
I20260619 12:03:50.094235 19405 raft_consensus.cc:348] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.094452 19405 raft_consensus.cc:374] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:50.094491 19405 raft_consensus.cc:729] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1986c5c8604b4a1fa8e2021ac6235ed7, State: Initialized, Role: FOLLOWER
I20260619 12:03:50.094610 19405 consensus_queue.cc:260] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.094923 19405 ts_tablet_manager.cc:1428] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:50.095119 19252 heartbeater.cc:503] Master 127.0.203.254:35459 was elected leader, sending a full tablet report...
I20260619 12:03:50.093765 19403 ts_tablet_manager.cc:1428] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:50.095885 19404 raft_consensus.cc:348] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.096120 19404 raft_consensus.cc:374] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:50.096163 19404 raft_consensus.cc:729] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28878ebbc4a940219e1766c7f700f4a1, State: Initialized, Role: FOLLOWER
I20260619 12:03:50.096279 19404 consensus_queue.cc:260] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.096661 19404 ts_tablet_manager.cc:1428] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260619 12:03:50.096683 19118 heartbeater.cc:503] Master 127.0.203.254:35459 was elected leader, sending a full tablet report...
I20260619 12:03:50.112134 19410 raft_consensus.cc:482] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:50.112316 19410 raft_consensus.cc:504] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.112962 19410 leader_election.cc:288] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28878ebbc4a940219e1766c7f700f4a1 (127.0.203.193:45409), 6290ec310452409390b36f0c06b00ec9 (127.0.203.195:46125)
I20260619 12:03:50.116485 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.113319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:55249 (local address 127.0.203.193:45409)
0619 12:03:50.113395 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.113398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.113407 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:03:50.113473 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.113474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.113486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.113566 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.113751 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.114504 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.115659 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.116065 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.116172 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.116187 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.116381 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.116424 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 12:03:50.116488 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.113186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:45409 (local address 127.0.203.194:55249)
0619 12:03:50.113257 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.113272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.113323 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.113570 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.113571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.113572 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.113665 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.113668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.114603 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.114604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.115562 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.115565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.116180 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.116181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.116272 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.116277 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.116343 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.116398 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 12:03:50.117055 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.113263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:46125 (local address 127.0.203.194:35651)
0619 12:03:50.113769 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.113782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.113882 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.114140 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.114141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.114145 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.114252 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.114255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.115050 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.115051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.115908 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.115911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.116780 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.116782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.116861 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.116866 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.116932 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.116976 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":149,"threads_started":1}
I20260619 12:03:50.117096 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.113503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:35651 (local address 127.0.203.195:46125)
0619 12:03:50.113566 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.113569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.113855 (+ 286us) server_negotiation.cc:407] Connection header received
0619 12:03:50.113987 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.113989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.114000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.114058 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.114373 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.114953 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.116005 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.116605 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.116713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.116724 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.116958 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.117028 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:03:50.117116 19072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb182e8b73b4b83859df3b675f5d52d" candidate_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28878ebbc4a940219e1766c7f700f4a1" is_pre_election: true
I20260619 12:03:50.117290 19072 raft_consensus.cc:2454] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1986c5c8604b4a1fa8e2021ac6235ed7 in term 0.
I20260619 12:03:50.117615 19140 leader_election.cc:302] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1986c5c8604b4a1fa8e2021ac6235ed7, 28878ebbc4a940219e1766c7f700f4a1; no voters:
I20260619 12:03:50.117726 19340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb182e8b73b4b83859df3b675f5d52d" candidate_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290ec310452409390b36f0c06b00ec9" is_pre_election: true
I20260619 12:03:50.117826 19410 raft_consensus.cc:2781] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:03:50.117909 19340 raft_consensus.cc:2454] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1986c5c8604b4a1fa8e2021ac6235ed7 in term 0.
I20260619 12:03:50.117949 19410 raft_consensus.cc:482] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:03:50.117981 19410 raft_consensus.cc:3037] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:50.118768 19410 raft_consensus.cc:504] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.118958 19410 leader_election.cc:288] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [CANDIDATE]: Term 1 election: Requested vote from peers 28878ebbc4a940219e1766c7f700f4a1 (127.0.203.193:45409), 6290ec310452409390b36f0c06b00ec9 (127.0.203.195:46125)
I20260619 12:03:50.119217 19340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb182e8b73b4b83859df3b675f5d52d" candidate_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290ec310452409390b36f0c06b00ec9"
I20260619 12:03:50.119310 19340 raft_consensus.cc:3037] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:50.119287 19072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eb182e8b73b4b83859df3b675f5d52d" candidate_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28878ebbc4a940219e1766c7f700f4a1"
I20260619 12:03:50.119377 19072 raft_consensus.cc:3037] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:50.120409 19340 raft_consensus.cc:2454] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1986c5c8604b4a1fa8e2021ac6235ed7 in term 1.
I20260619 12:03:50.120409 19072 raft_consensus.cc:2454] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1986c5c8604b4a1fa8e2021ac6235ed7 in term 1.
I20260619 12:03:50.120697 19141 leader_election.cc:302] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1986c5c8604b4a1fa8e2021ac6235ed7, 6290ec310452409390b36f0c06b00ec9; no voters:
I20260619 12:03:50.120803 19410 raft_consensus.cc:2781] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:50.121023 19410 raft_consensus.cc:686] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 1986c5c8604b4a1fa8e2021ac6235ed7, State: Running, Role: LEADER
I20260619 12:03:50.121131 19410 consensus_queue.cc:237] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } }
I20260619 12:03:50.122151 18927 catalog_manager.cc:5495] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1986c5c8604b4a1fa8e2021ac6235ed7 (127.0.203.194). New cstate: current_term: 1 leader_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1986c5c8604b4a1fa8e2021ac6235ed7" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 40477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 } health_report { overall_health: UNKNOWN } } }
I20260619 12:03:50.129334 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.126379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.0.203.194:40477)
0619 12:03:50.126436 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.126438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.126458 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:03:50.126563 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.126564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.126581 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.126667 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:50.126895 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.127394 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.128277 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.128478 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.128559 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.128886 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:50.129000 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:03:50.129103 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.129260 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.129270 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 12:03:50.131415 19340 raft_consensus.cc:1261] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 1 FOLLOWER]: Refusing update from remote peer 1986c5c8604b4a1fa8e2021ac6235ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:50.131433 19072 raft_consensus.cc:1261] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 1 FOLLOWER]: Refusing update from remote peer 1986c5c8604b4a1fa8e2021ac6235ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:50.131880 19413 consensus_queue.cc:1035] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28878ebbc4a940219e1766c7f700f4a1" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:50.132040 19410 consensus_queue.cc:1035] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6290ec310452409390b36f0c06b00ec9" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 46125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:50.135545 19418 mvcc.cc:205] Tried to move back new op lower bound from 7298542101014798336 to 7298542100976844800. Current Snapshot: MvccSnapshot[applied={T|T < 7298542101014798336}]
I20260619 12:03:50.135886 19417 mvcc.cc:205] Tried to move back new op lower bound from 7298542101014798336 to 7298542100976844800. Current Snapshot: MvccSnapshot[applied={T|T < 7298542101014798336}]
I20260619 12:03:50.138638 18927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:50.138778 18927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:50.140048 18927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36caab4a3c9e4eac8c8010ff12091d0d: Sending DeleteTablet for 3 replicas of tablet 2eb182e8b73b4b83859df3b675f5d52d
I20260619 12:03:50.140522 19052 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb182e8b73b4b83859df3b675f5d52d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:50 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:60590
I20260619 12:03:50.140672 19320 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb182e8b73b4b83859df3b675f5d52d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:50 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:39288
I20260619 12:03:50.140709 19186 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eb182e8b73b4b83859df3b675f5d52d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:50 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:41980
I20260619 12:03:50.140916 19426 tablet_replica.cc:330] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1: stopping tablet replica
I20260619 12:03:50.141062 19426 raft_consensus.cc:2229] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:50.141206 19426 raft_consensus.cc:2258] T 2eb182e8b73b4b83859df3b675f5d52d P 28878ebbc4a940219e1766c7f700f4a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:50.141409 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18989
I20260619 12:03:50.142212 19427 tablet_replica.cc:330] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: stopping tablet replica
I20260619 12:03:50.142344 19427 raft_consensus.cc:2229] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:50.142462 19427 raft_consensus.cc:2258] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:50.142926 19427 ts_tablet_manager.cc:1905] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:50.144604 19427 ts_tablet_manager.cc:1918] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:50.144742 19427 log.cc:1196] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/2eb182e8b73b4b83859df3b675f5d52d
I20260619 12:03:50.144960 19428 tablet_replica.cc:330] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: stopping tablet replica
I20260619 12:03:50.145054 19427 ts_tablet_manager.cc:1939] T 2eb182e8b73b4b83859df3b675f5d52d P 6290ec310452409390b36f0c06b00ec9: Deleting consensus metadata
I20260619 12:03:50.145336 19428 raft_consensus.cc:2229] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:03:50.145550 18913 catalog_manager.cc:4841] TS 6290ec310452409390b36f0c06b00ec9 (127.0.203.195:46125): tablet 2eb182e8b73b4b83859df3b675f5d52d (table test-table [id=dbed73b0a73c4b6d887a74d2eb94c2c2]) successfully deleted
I20260619 12:03:50.145941 19428 raft_consensus.cc:2258] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:50.146396 19428 ts_tablet_manager.cc:1905] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:50.148000 19428 ts_tablet_manager.cc:1918] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:50.148136 19428 log.cc:1196] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/2eb182e8b73b4b83859df3b675f5d52d
I20260619 12:03:50.148434 19428 ts_tablet_manager.cc:1939] T 2eb182e8b73b4b83859df3b675f5d52d P 1986c5c8604b4a1fa8e2021ac6235ed7: Deleting consensus metadata
I20260619 12:03:50.148986 18913 catalog_manager.cc:4841] TS 1986c5c8604b4a1fa8e2021ac6235ed7 (127.0.203.194:40477): tablet 2eb182e8b73b4b83859df3b675f5d52d (table test-table [id=dbed73b0a73c4b6d887a74d2eb94c2c2]) successfully deleted
W20260619 12:03:50.150844 18912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:45409 (error 108)
I20260619 12:03:50.151067 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 19123
I20260619 12:03:50.152371 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.152221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:45409 (local address 127.0.0.1:60600)
0619 12:03:50.152291 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.152330 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:45409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:03:50.152580 18912 catalog_manager.cc:4584] TS 28878ebbc4a940219e1766c7f700f4a1 (127.0.203.193:45409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2eb182e8b73b4b83859df3b675f5d52d: Network error: Client connection negotiation failed: client connection to 127.0.203.193:45409: connect: Connection refused (error 111)
I20260619 12:03:50.157155 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 19257
I20260619 12:03:50.162511 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 18895
2026-06-19T12:03:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2088 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T12:03:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:50Z Disabled control of system clock
I20260619 12:03:50.194526 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:39237
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:39237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:03:50.287297 19433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:50.287539 19433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:50.287560 19433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:50.288717 19433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:50.288758 19433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:50.288771 19433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:50.288784 19433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:50.288797 19433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:50.290341 19433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:39237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:39237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:50.290572 19433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:50.290843 19433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:50.296903 19439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:50.297009 19438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:50.297045 19441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:50.297511 19433 server_base.cc:1034] running on GCE node
I20260619 12:03:50.297825 19433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:50.298218 19433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:50.299441 19433 hybrid_clock.cc:648] HybridClock initialized: now 1781870630299421 us; error 37 us; skew 500 ppm
I20260619 12:03:50.301098 19433 webserver.cc:460] Webserver started at http://127.0.203.254:33193/ using document root <none> and password file <none>
I20260619 12:03:50.301384 19433 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:50.301458 19433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:50.301625 19433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:50.303027 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "18d59d63b1b24bf99ccf47626f996573"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "9e55cdf02872ed982c081402d30b0f8c"
server_key_iv: "94ac2e360b647b22126ce4b3d3d078a4"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.303447 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "18d59d63b1b24bf99ccf47626f996573"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "9e55cdf02872ed982c081402d30b0f8c"
server_key_iv: "94ac2e360b647b22126ce4b3d3d078a4"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.305181 19433 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 12:03:50.306330 19447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.306675 19433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:50.306829 19433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "18d59d63b1b24bf99ccf47626f996573"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "9e55cdf02872ed982c081402d30b0f8c"
server_key_iv: "94ac2e360b647b22126ce4b3d3d078a4"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.306998 19433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:50.322494 19433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:50.322911 19433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:50.323078 19433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:50.328338 19433 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:39237
I20260619 12:03:50.328410 19499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:39237 every 8 connection(s)
I20260619 12:03:50.328836 19433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:50.329974 19500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:50.330687 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 19433
I20260619 12:03:50.330821 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:50.331068 815 external_mini_cluster.cc:1428] Setting key b47fe7da0258c7b206223e28f92125a6
I20260619 12:03:50.333629 19500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573: Bootstrap starting.
I20260619 12:03:50.334311 19500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:50.334657 19500 log.cc:824] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:50.335477 19500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573: No bootstrap required, opened a new log
I20260619 12:03:50.337575 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.331732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.0.203.254:39237)
0619 12:03:50.331961 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.331970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.332001 (+ 31us) server_negotiation.cc:407] Connection header received
0619 12:03:50.332579 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.332587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.332749 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.332894 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:50.333221 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.334205 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.335284 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.335825 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.335941 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.336816 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:50.336820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:50.336821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:50.337007 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:50.337044 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:50.337056 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:50.337113 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.337235 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.337342 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260619 12:03:50.338061 19500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d59d63b1b24bf99ccf47626f996573" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39237 } }
I20260619 12:03:50.338240 19500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:50.338267 19500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d59d63b1b24bf99ccf47626f996573, State: Initialized, Role: FOLLOWER
I20260619 12:03:50.338402 19500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d59d63b1b24bf99ccf47626f996573" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39237 } }
I20260619 12:03:50.338487 19500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:50.338517 19500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:50.338594 19500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:50.339226 19500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d59d63b1b24bf99ccf47626f996573" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39237 } }
I20260619 12:03:50.339315 19500 leader_election.cc:302] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d59d63b1b24bf99ccf47626f996573; no voters:
I20260619 12:03:50.339462 19500 leader_election.cc:288] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:50.339586 19505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:50.339790 19505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [term 1 LEADER]: Becoming Leader. State: Replica: 18d59d63b1b24bf99ccf47626f996573, State: Running, Role: LEADER
I20260619 12:03:50.339812 19500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:50.339908 19505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d59d63b1b24bf99ccf47626f996573" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39237 } }
I20260619 12:03:50.340405 19506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d59d63b1b24bf99ccf47626f996573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d59d63b1b24bf99ccf47626f996573" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39237 } } }
I20260619 12:03:50.340436 19507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d59d63b1b24bf99ccf47626f996573. Latest consensus state: current_term: 1 leader_uuid: "18d59d63b1b24bf99ccf47626f996573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d59d63b1b24bf99ccf47626f996573" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 39237 } } }
I20260619 12:03:50.340523 19506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:50.340531 19507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:50.340833 19514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:50.341513 19514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:50.343534 19514 catalog_manager.cc:1347] Generated new cluster ID: 59dc272885fa4880a6df41c6f341d349
I20260619 12:03:50.343603 19514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:50.358501 19514 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:50.359411 19514 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:50.366024 19514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573: Generated new TSK 0
I20260619 12:03:50.366254 19514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:03:50.369014 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39237
--builtin_ntp_servers=127.0.203.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:03:50.462954 19524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:50.463222 19524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:50.463258 19524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:50.464484 19524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:50.464543 19524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:50.464587 19524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:50.465986 19524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:50.466260 19524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:50.466569 19524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:50.472124 19532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:50.472050 19529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:50.472193 19530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:50.472604 19524 server_base.cc:1034] running on GCE node
I20260619 12:03:50.472842 19524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:50.473179 19524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:50.474431 19524 hybrid_clock.cc:648] HybridClock initialized: now 1781870630474417 us; error 31 us; skew 500 ppm
I20260619 12:03:50.475890 19524 webserver.cc:460] Webserver started at http://127.0.203.193:41993/ using document root <none> and password file <none>
I20260619 12:03:50.476163 19524 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:50.476238 19524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:50.476402 19524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:50.477695 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "6492770add75491fb092c9958a5d5827"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "2e99ba4502e8ccebfaf93a95b7a6b5a4"
server_key_iv: "18075d92085d68921f1bcc337c0b96ac"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.478091 19524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "6492770add75491fb092c9958a5d5827"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "2e99ba4502e8ccebfaf93a95b7a6b5a4"
server_key_iv: "18075d92085d68921f1bcc337c0b96ac"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.479611 19524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:50.480527 19538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.480867 19524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:50.481005 19524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "6492770add75491fb092c9958a5d5827"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "2e99ba4502e8ccebfaf93a95b7a6b5a4"
server_key_iv: "18075d92085d68921f1bcc337c0b96ac"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.481118 19524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:50.493096 19524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:50.493458 19524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:50.493625 19524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:50.493901 19524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:50.494282 19524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:50.494341 19524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.494410 19524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:50.494462 19524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.502571 19524 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:41839
I20260619 12:03:50.502709 19651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:41839 every 8 connection(s)
I20260619 12:03:50.503110 19524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:03:50.505223 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 19524
I20260619 12:03:50.505338 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:03:50.505607 815 external_mini_cluster.cc:1428] Setting key 04b3906f28c2e6c1d0d310bf9d8c9f8e
I20260619 12:03:50.507025 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39237
--builtin_ntp_servers=127.0.203.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:03:50.510923 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.504152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:41559 (local address 127.0.203.254:39237)
0619 12:03:50.504306 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.504318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.504730 (+ 412us) server_negotiation.cc:407] Connection header received
0619 12:03:50.505471 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.505472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.505495 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.505563 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:50.506665 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.507519 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.508806 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.509038 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.509124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.510174 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:50.510176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:50.510176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:50.510182 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:50.510200 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:50.510205 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:50.510261 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.510760 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.510827 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:03:50.511255 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.504118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39237 (local address 127.0.203.193:41559)
0619 12:03:50.504493 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.504535 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.505336 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.505670 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.505675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.505694 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:50.506468 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.506474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.507656 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.507657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.508570 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.508572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.509148 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.509148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.509313 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.510005 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:50.510006 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:50.510040 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:50.510070 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:50.510308 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:50.510314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:50.510317 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.510760 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.510953 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":217,"threads_started":1}
I20260619 12:03:50.512353 19652 heartbeater.cc:347] Connected to a master server at 127.0.203.254:39237
I20260619 12:03:50.512494 19652 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:50.512799 19652 heartbeater.cc:511] Master 127.0.203.254:39237 requested a full tablet report, sending...
I20260619 12:03:50.513549 19464 ts_manager.cc:194] Registered new tserver with Master: 6492770add75491fb092c9958a5d5827 (127.0.203.193:41839)
I20260619 12:03:50.514276 19464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.193:41559
W20260619 12:03:50.597716 19655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:50.597963 19655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:50.597988 19655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:50.599253 19655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:50.599301 19655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:50.599326 19655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:50.600742 19655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:50.601016 19655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:50.601318 19655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:50.606832 19660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:50.607052 19655 server_base.cc:1034] running on GCE node
W20260619 12:03:50.606895 19663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:50.606817 19661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:50.607456 19655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:50.607820 19655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:50.609002 19655 hybrid_clock.cc:648] HybridClock initialized: now 1781870630608993 us; error 57 us; skew 500 ppm
I20260619 12:03:50.610670 19655 webserver.cc:460] Webserver started at http://127.0.203.194:39613/ using document root <none> and password file <none>
I20260619 12:03:50.610993 19655 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:50.611078 19655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:50.611236 19655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:50.612486 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "b1452a7c9afc4630a8177b27e5d2f99d"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "f7ec83943338fd269ceb6aad7488ae53"
server_key_iv: "8d234518bd40ba84b7d7b7691045d632"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.612921 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "b1452a7c9afc4630a8177b27e5d2f99d"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "f7ec83943338fd269ceb6aad7488ae53"
server_key_iv: "8d234518bd40ba84b7d7b7691045d632"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.614429 19655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:50.615550 19669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.615885 19655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:50.616014 19655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "b1452a7c9afc4630a8177b27e5d2f99d"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "f7ec83943338fd269ceb6aad7488ae53"
server_key_iv: "8d234518bd40ba84b7d7b7691045d632"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.616109 19655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:50.639427 19655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:50.639869 19655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:50.640132 19655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:50.640426 19655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:50.640882 19655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:50.640920 19655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.640946 19655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:50.640959 19655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.648881 19655 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:39249
I20260619 12:03:50.649113 19782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:39249 every 8 connection(s)
I20260619 12:03:50.649394 19655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:03:50.653208 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 19655
I20260619 12:03:50.653337 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:03:50.653576 815 external_mini_cluster.cc:1428] Setting key ddc6a9be1912d70cb6c140875ea28479
I20260619 12:03:50.654772 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39237
--builtin_ntp_servers=127.0.203.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:03:50.655561 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.650350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:43875 (local address 127.0.203.254:39237)
0619 12:03:50.650434 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.650437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.650736 (+ 299us) server_negotiation.cc:407] Connection header received
0619 12:03:50.651351 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.651352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.651365 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.651415 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:50.652150 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.652732 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.653844 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.654115 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.654201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.655265 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:50.655266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:50.655267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:50.655272 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:50.655287 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:50.655292 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:50.655340 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.655429 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.655464 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:03:50.655782 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.650341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39237 (local address 127.0.203.194:43875)
0619 12:03:50.650589 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.650627 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.651236 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.651487 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.651495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.651503 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:50.652041 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.652045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.652831 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.652832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.653759 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.653761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.654208 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.654210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.654352 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.655114 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:50.655114 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:50.655144 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:50.655169 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:50.655339 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:50.655344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:50.655347 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.655432 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.655540 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
I20260619 12:03:50.656399 19783 heartbeater.cc:347] Connected to a master server at 127.0.203.254:39237
I20260619 12:03:50.656510 19783 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:50.656782 19783 heartbeater.cc:511] Master 127.0.203.254:39237 requested a full tablet report, sending...
I20260619 12:03:50.657207 19464 ts_manager.cc:194] Registered new tserver with Master: b1452a7c9afc4630a8177b27e5d2f99d (127.0.203.194:39249)
I20260619 12:03:50.657614 19464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.194:43875
W20260619 12:03:50.743168 19786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:50.743410 19786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:50.743435 19786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:50.744884 19786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:50.744954 19786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:50.745018 19786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:50.746678 19786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:50.746978 19786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:50.747346 19786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:50.753185 19791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:50.753302 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:50.753322 19794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:50.753587 19786 server_base.cc:1034] running on GCE node
I20260619 12:03:50.753844 19786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:50.754108 19786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:50.755402 19786 hybrid_clock.cc:648] HybridClock initialized: now 1781870630755375 us; error 54 us; skew 500 ppm
I20260619 12:03:50.757431 19786 webserver.cc:460] Webserver started at http://127.0.203.195:32895/ using document root <none> and password file <none>
I20260619 12:03:50.757720 19786 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:50.757799 19786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:50.757970 19786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:50.759117 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "245a9c3863944bc1909e91a6cb95b66e"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "29b19b9062c19657bc3a6af731880820"
server_key_iv: "46e517fd120aec6a8c19c3e2f06418b2"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.759512 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "245a9c3863944bc1909e91a6cb95b66e"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "29b19b9062c19657bc3a6af731880820"
server_key_iv: "46e517fd120aec6a8c19c3e2f06418b2"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.761116 19786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260619 12:03:50.762058 19800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.762343 19786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:50.762457 19786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "245a9c3863944bc1909e91a6cb95b66e"
format_stamp: "Formatted at 2026-06-19 12:03:50 on dist-test-slave-fbq8"
server_key: "29b19b9062c19657bc3a6af731880820"
server_key_iv: "46e517fd120aec6a8c19c3e2f06418b2"
server_key_version: "encryptionkey@0"
I20260619 12:03:50.762553 19786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:50.775442 19786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:50.775799 19786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:50.775964 19786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:50.776255 19786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:50.776711 19786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:50.776779 19786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.776824 19786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:50.776854 19786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:50.783862 19786 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:44893
I20260619 12:03:50.784037 19913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:44893 every 8 connection(s)
I20260619 12:03:50.784323 19786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:03:50.791560 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 19786
I20260619 12:03:50.791699 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:03:50.791929 815 external_mini_cluster.cc:1428] Setting key 039bb1ba48ebbc7d961040dd1ba2220a
I20260619 12:03:50.791923 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.785634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:36837 (local address 127.0.203.254:39237)
0619 12:03:50.785716 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.785718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.786211 (+ 493us) server_negotiation.cc:407] Connection header received
0619 12:03:50.786780 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.786781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.786793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.786880 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:50.788068 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.788582 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.789793 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.790036 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.790119 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.791532 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:50.791534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:50.791535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:50.791545 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:50.791571 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:50.791576 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:50.791658 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.791793 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.791831 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:03:50.792205 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.785739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:39237 (local address 127.0.203.195:36837)
0619 12:03:50.786006 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.786058 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.786679 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.786975 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.786979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.786989 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:50.787757 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.787768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.788751 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.788754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.789674 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.789678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.790199 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.790200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.790437 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.791132 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:50.791132 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:50.791169 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:50.791203 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:50.791660 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:50.791668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:50.791672 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.791760 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.791901 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":146,"threads_started":1}
I20260619 12:03:50.792943 19914 heartbeater.cc:347] Connected to a master server at 127.0.203.254:39237
I20260619 12:03:50.793056 19914 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:50.793288 19914 heartbeater.cc:511] Master 127.0.203.254:39237 requested a full tablet report, sending...
I20260619 12:03:50.793780 19464 ts_manager.cc:194] Registered new tserver with Master: 245a9c3863944bc1909e91a6cb95b66e (127.0.203.195:44893)
I20260619 12:03:50.794186 19464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.195:36837
I20260619 12:03:50.804737 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:03:50.809938 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.806429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.0.203.254:39237)
0619 12:03:50.806522 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.806524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.806685 (+ 161us) server_negotiation.cc:407] Connection header received
0619 12:03:50.806720 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.806721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.806732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.806799 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:50.807017 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.807539 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.808391 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.808589 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.808771 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.809544 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:50.809545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:50.809546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:50.809551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:50.809568 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:50.809573 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:50.809656 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.809798 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.809811 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:03:50.812069 19464 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:03:50.813777 19464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:03:50.821203 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.816791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.0.203.194:39249)
0619 12:03:50.817068 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.817088 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.817104 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:03:50.817158 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.817160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.817177 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.817266 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.817531 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.818740 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.820107 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.820754 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.820846 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.820957 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.821039 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.821088 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20260619 12:03:50.821336 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.816409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44893 (local address 127.0.0.1:55396)
0619 12:03:50.816548 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.816589 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.816730 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.817490 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.817491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.817494 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.817566 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.817568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.818517 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.818518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.819954 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.819957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.820980 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.820982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.821059 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.821066 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.821099 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.821162 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":34,"thread_start_us":84,"threads_started":1}
I20260619 12:03:50.821517 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.816791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.0.203.195:44893)
0619 12:03:50.817283 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.817289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.817302 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:03:50.817351 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.817353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.817372 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.817648 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.817697 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.818424 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.820082 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.820867 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.821237 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.821344 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.821373 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.821419 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":112,"threads_started":1}
I20260619 12:03:50.821839 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.816418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41839 (local address 127.0.0.1:48028)
0619 12:03:50.817005 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.817018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.817091 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.817354 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.817358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.817363 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.817445 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.817447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.818841 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.818842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.819971 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.819973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.821582 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.821583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.821652 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.821656 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.821707 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.821759 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":9,"thread_start_us":93,"threads_started":1}
I20260619 12:03:50.821189 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.816420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:39249 (local address 127.0.0.1:47088)
0619 12:03:50.817005 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.817018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.817087 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.817354 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.817358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.817361 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.817445 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.817447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.818836 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.818837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.819999 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.820001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.820844 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.820845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.820922 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.820935 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.821017 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.821088 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":107,"threads_started":1}
I20260619 12:03:50.822270 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.816765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.0.203.193:41839)
0619 12:03:50.816975 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.816982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.817105 (+ 123us) server_negotiation.cc:407] Connection header received
0619 12:03:50.817158 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.817160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.817177 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.817266 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.817530 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.818734 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.820719 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.821404 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.821469 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.821544 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.822150 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.822192 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":133,"threads_started":1}
I20260619 12:03:50.822405 19848 tablet_service.cc:1467] Processing CreateTablet for tablet dbe5aa9a289e4c728ed7d81911038b3d (DEFAULT_TABLE table=test-table [id=533e5e68d21340d2b373b216bf3705bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:50.822793 19848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe5aa9a289e4c728ed7d81911038b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:50.822865 19586 tablet_service.cc:1467] Processing CreateTablet for tablet dbe5aa9a289e4c728ed7d81911038b3d (DEFAULT_TABLE table=test-table [id=533e5e68d21340d2b373b216bf3705bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:50.823194 19586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe5aa9a289e4c728ed7d81911038b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:50.822971 19717 tablet_service.cc:1467] Processing CreateTablet for tablet dbe5aa9a289e4c728ed7d81911038b3d (DEFAULT_TABLE table=test-table [id=533e5e68d21340d2b373b216bf3705bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:50.823549 19717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe5aa9a289e4c728ed7d81911038b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:50.826628 19928 tablet_bootstrap.cc:492] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827: Bootstrap starting.
I20260619 12:03:50.827096 19930 tablet_bootstrap.cc:492] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Bootstrap starting.
I20260619 12:03:50.827646 19928 tablet_bootstrap.cc:654] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:50.827927 19930 tablet_bootstrap.cc:654] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:50.828027 19928 log.cc:824] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:50.828269 19930 log.cc:824] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:50.828682 19929 tablet_bootstrap.cc:492] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Bootstrap starting.
I20260619 12:03:50.829114 19928 tablet_bootstrap.cc:492] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827: No bootstrap required, opened a new log
I20260619 12:03:50.829192 19928 ts_tablet_manager.cc:1397] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:03:50.829401 19929 tablet_bootstrap.cc:654] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:50.829883 19929 log.cc:824] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:50.830607 19929 tablet_bootstrap.cc:492] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: No bootstrap required, opened a new log
I20260619 12:03:50.830696 19929 ts_tablet_manager.cc:1397] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:03:50.831496 19928 raft_consensus.cc:348] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.831670 19928 raft_consensus.cc:374] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:50.831696 19928 raft_consensus.cc:729] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6492770add75491fb092c9958a5d5827, State: Initialized, Role: FOLLOWER
I20260619 12:03:50.831827 19928 consensus_queue.cc:260] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.832312 19652 heartbeater.cc:503] Master 127.0.203.254:39237 was elected leader, sending a full tablet report...
I20260619 12:03:50.832325 19928 ts_tablet_manager.cc:1428] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:50.832799 19930 tablet_bootstrap.cc:492] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: No bootstrap required, opened a new log
I20260619 12:03:50.832895 19930 ts_tablet_manager.cc:1397] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 12:03:50.833523 19929 raft_consensus.cc:348] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.833699 19929 raft_consensus.cc:374] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:50.833745 19929 raft_consensus.cc:729] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245a9c3863944bc1909e91a6cb95b66e, State: Initialized, Role: FOLLOWER
I20260619 12:03:50.834017 19929 consensus_queue.cc:260] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.834403 19914 heartbeater.cc:503] Master 127.0.203.254:39237 was elected leader, sending a full tablet report...
I20260619 12:03:50.834712 19929 ts_tablet_manager.cc:1428] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260619 12:03:50.835615 19930 raft_consensus.cc:348] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.835830 19930 raft_consensus.cc:374] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:50.835868 19930 raft_consensus.cc:729] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1452a7c9afc4630a8177b27e5d2f99d, State: Initialized, Role: FOLLOWER
I20260619 12:03:50.836006 19930 consensus_queue.cc:260] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.836266 19930 ts_tablet_manager.cc:1428] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 12:03:50.836434 19783 heartbeater.cc:503] Master 127.0.203.254:39237 was elected leader, sending a full tablet report...
I20260619 12:03:50.868263 19934 raft_consensus.cc:482] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:50.868399 19934 raft_consensus.cc:504] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.868889 19934 leader_election.cc:288] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1452a7c9afc4630a8177b27e5d2f99d (127.0.203.194:39249), 245a9c3863944bc1909e91a6cb95b66e (127.0.203.195:44893)
I20260619 12:03:50.872152 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.869041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:39249 (local address 127.0.203.193:49559)
0619 12:03:50.869105 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.869121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.869179 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.869352 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.869353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.869356 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.869468 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.869471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.870327 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.870328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.871217 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.871220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.871881 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.871882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.871961 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.871967 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.872031 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.872091 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":25}
I20260619 12:03:50.872231 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.869137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:49559 (local address 127.0.203.194:39249)
0619 12:03:50.869193 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.869195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.869210 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:03:50.869247 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.869249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.869263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.869337 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.869608 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.870210 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.871321 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.871771 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.871834 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.871844 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.872056 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.872113 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:03:50.872346 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.869140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:44893 (local address 127.0.203.193:42841)
0619 12:03:50.869365 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.869372 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:50.869427 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:50.869600 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:50.869600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:50.869603 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:50.869697 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.869699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.870502 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.870503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.871375 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:50.871377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.872060 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.872061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:50.872144 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.872149 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:03:50.872210 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:03:50.872254 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":2,"thread_start_us":138,"threads_started":1}
I20260619 12:03:50.872385 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.869228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:42841 (local address 127.0.203.195:44893)
0619 12:03:50.869276 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.869278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.869449 (+ 171us) server_negotiation.cc:407] Connection header received
0619 12:03:50.869490 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.869491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.869506 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.869572 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:50.869817 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.870406 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.871499 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.871928 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.872026 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.872035 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.872237 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.872287 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:03:50.872677 19868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe5aa9a289e4c728ed7d81911038b3d" candidate_uuid: "6492770add75491fb092c9958a5d5827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245a9c3863944bc1909e91a6cb95b66e" is_pre_election: true
I20260619 12:03:50.872673 19737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe5aa9a289e4c728ed7d81911038b3d" candidate_uuid: "6492770add75491fb092c9958a5d5827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" is_pre_election: true
I20260619 12:03:50.872848 19868 raft_consensus.cc:2454] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6492770add75491fb092c9958a5d5827 in term 0.
I20260619 12:03:50.872848 19737 raft_consensus.cc:2454] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6492770add75491fb092c9958a5d5827 in term 0.
I20260619 12:03:50.873135 19542 leader_election.cc:302] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245a9c3863944bc1909e91a6cb95b66e, 6492770add75491fb092c9958a5d5827; no voters:
I20260619 12:03:50.873323 19934 raft_consensus.cc:2781] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:03:50.873414 19934 raft_consensus.cc:482] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:03:50.873452 19934 raft_consensus.cc:3037] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:50.874204 19934 raft_consensus.cc:504] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.874380 19934 leader_election.cc:288] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [CANDIDATE]: Term 1 election: Requested vote from peers b1452a7c9afc4630a8177b27e5d2f99d (127.0.203.194:39249), 245a9c3863944bc1909e91a6cb95b66e (127.0.203.195:44893)
I20260619 12:03:50.874554 19868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe5aa9a289e4c728ed7d81911038b3d" candidate_uuid: "6492770add75491fb092c9958a5d5827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245a9c3863944bc1909e91a6cb95b66e"
I20260619 12:03:50.874562 19737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe5aa9a289e4c728ed7d81911038b3d" candidate_uuid: "6492770add75491fb092c9958a5d5827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1452a7c9afc4630a8177b27e5d2f99d"
I20260619 12:03:50.874642 19868 raft_consensus.cc:3037] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:50.874694 19737 raft_consensus.cc:3037] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:50.875372 19868 raft_consensus.cc:2454] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6492770add75491fb092c9958a5d5827 in term 1.
I20260619 12:03:50.875602 19542 leader_election.cc:302] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245a9c3863944bc1909e91a6cb95b66e, 6492770add75491fb092c9958a5d5827; no voters:
I20260619 12:03:50.875697 19934 raft_consensus.cc:2781] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:50.875759 19737 raft_consensus.cc:2454] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6492770add75491fb092c9958a5d5827 in term 1.
I20260619 12:03:50.875908 19934 raft_consensus.cc:686] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 1 LEADER]: Becoming Leader. State: Replica: 6492770add75491fb092c9958a5d5827, State: Running, Role: LEADER
I20260619 12:03:50.876021 19934 consensus_queue.cc:237] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } }
I20260619 12:03:50.877104 19463 catalog_manager.cc:5495] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6492770add75491fb092c9958a5d5827 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "6492770add75491fb092c9958a5d5827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6492770add75491fb092c9958a5d5827" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 41839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 } health_report { overall_health: UNKNOWN } } }
I20260619 12:03:50.883390 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.880680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.0.203.193:41839)
0619 12:03:50.880732 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:50.880734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:50.880744 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:03:50.880773 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:50.880773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:50.880791 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:50.880866 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:50.881113 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.881579 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.882443 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:50.882640 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:50.882710 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:50.883014 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:50.883117 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:03:50.883225 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:50.883293 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 12:03:50.883303 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:03:50.885735 19868 raft_consensus.cc:1261] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 1 FOLLOWER]: Refusing update from remote peer 6492770add75491fb092c9958a5d5827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:50.885794 19737 raft_consensus.cc:1261] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 1 FOLLOWER]: Refusing update from remote peer 6492770add75491fb092c9958a5d5827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:50.886265 19934 consensus_queue.cc:1035] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245a9c3863944bc1909e91a6cb95b66e" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:50.886438 19938 consensus_queue.cc:1035] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1452a7c9afc4630a8177b27e5d2f99d" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:50.889336 19941 mvcc.cc:205] Tried to move back new op lower bound from 7298542104105013248 to 7298542104069218304. Current Snapshot: MvccSnapshot[applied={T|T < 7298542104105013248}]
I20260619 12:03:50.892305 19464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:50.892441 19464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:50.893463 19464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18d59d63b1b24bf99ccf47626f996573: Sending DeleteTablet for 3 replicas of tablet dbe5aa9a289e4c728ed7d81911038b3d
I20260619 12:03:50.893901 19586 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe5aa9a289e4c728ed7d81911038b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:50 UTC) from {username='slave'} at 127.0.0.1:48028
I20260619 12:03:50.893903 19717 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe5aa9a289e4c728ed7d81911038b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:50 UTC) from {username='slave'} at 127.0.0.1:47088
I20260619 12:03:50.894327 19950 tablet_replica.cc:330] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827: stopping tablet replica
I20260619 12:03:50.894436 19950 raft_consensus.cc:2229] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:03:50.894400 19848 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe5aa9a289e4c728ed7d81911038b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:50 UTC) from {username='slave'} at 127.0.0.1:55396
I20260619 12:03:50.894541 19950 raft_consensus.cc:2258] T dbe5aa9a289e4c728ed7d81911038b3d P 6492770add75491fb092c9958a5d5827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:50.894716 19951 tablet_replica.cc:330] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: stopping tablet replica
I20260619 12:03:50.894761 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 19524
I20260619 12:03:50.894832 19951 raft_consensus.cc:2229] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:50.895583 19951 raft_consensus.cc:2258] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:50.896130 19951 ts_tablet_manager.cc:1905] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:50.896560 19952 tablet_replica.cc:330] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: stopping tablet replica
I20260619 12:03:50.896736 19952 raft_consensus.cc:2229] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:50.896871 19952 raft_consensus.cc:2258] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:50.897432 19952 ts_tablet_manager.cc:1905] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:50.898078 19951 ts_tablet_manager.cc:1918] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:50.898169 19951 log.cc:1196] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/dbe5aa9a289e4c728ed7d81911038b3d
I20260619 12:03:50.898473 19951 ts_tablet_manager.cc:1939] T dbe5aa9a289e4c728ed7d81911038b3d P b1452a7c9afc4630a8177b27e5d2f99d: Deleting consensus metadata
I20260619 12:03:50.899065 19450 catalog_manager.cc:4841] TS b1452a7c9afc4630a8177b27e5d2f99d (127.0.203.194:39249): tablet dbe5aa9a289e4c728ed7d81911038b3d (table test-table [id=533e5e68d21340d2b373b216bf3705bb]) successfully deleted
I20260619 12:03:50.900310 19952 ts_tablet_manager.cc:1918] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:50.900401 19952 log.cc:1196] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/dbe5aa9a289e4c728ed7d81911038b3d
I20260619 12:03:50.900705 19952 ts_tablet_manager.cc:1939] T dbe5aa9a289e4c728ed7d81911038b3d P 245a9c3863944bc1909e91a6cb95b66e: Deleting consensus metadata
I20260619 12:03:50.901348 19451 catalog_manager.cc:4841] TS 245a9c3863944bc1909e91a6cb95b66e (127.0.203.195:44893): tablet dbe5aa9a289e4c728ed7d81911038b3d (table test-table [id=533e5e68d21340d2b373b216bf3705bb]) successfully deleted
W20260619 12:03:50.904007 19449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:41839 (error 108)
I20260619 12:03:50.904186 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 19655
I20260619 12:03:50.905076 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:50.904919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:41839 (local address 127.0.0.1:48044)
0619 12:03:50.904994 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:50.905039 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:03:50.905226 19449 catalog_manager.cc:4584] TS 6492770add75491fb092c9958a5d5827 (127.0.203.193:41839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbe5aa9a289e4c728ed7d81911038b3d: Network error: Client connection negotiation failed: client connection to 127.0.203.193:41839: connect: Connection refused (error 111)
I20260619 12:03:50.909796 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 19786
I20260619 12:03:50.915979 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 19433
2026-06-19T12:03:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (752 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:50 dist-test-slave-fbq8 krb5kdc[19956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:50 dist-test-slave-fbq8 krb5kdc[19956](info): set up 2 sockets
Jun 19 12:03:50 dist-test-slave-fbq8 krb5kdc[19956](info): commencing operation
krb5kdc: starting...
W20260619 12:03:51.992825 815 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:03:52.133024 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:43583
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:52.221105 19969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:52.221412 19969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:52.221459 19969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:52.222815 19969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:52.222913 19969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:52.222944 19969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:52.222970 19969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:52.222997 19969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:52.224740 19969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:43583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:52.225069 19969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:52.225327 19969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:03:52.230861 19969 server_base.cc:1034] running on GCE node
W20260619 12:03:52.230894 19977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:52.231015 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:52.230880 19974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:52.231756 19969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:52.232115 19969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:52.233294 19969 hybrid_clock.cc:648] HybridClock initialized: now 1781870632233240 us; error 76 us; skew 500 ppm
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:52.236305 19969 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:03:52.236929 19969 webserver.cc:460] Webserver started at http://127.0.203.254:42751/ using document root <none> and password file <none>
I20260619 12:03:52.237196 19969 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:52.237253 19969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:52.237406 19969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:52.238617 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "a402d3f2b5364a93afae675d51ee78b5"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "8a08cc7aafa5ac833cba7af67b7e347c"
server_key_iv: "4bac5a3ad344041ea9b7a9ea2b13117d"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.238991 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "a402d3f2b5364a93afae675d51ee78b5"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "8a08cc7aafa5ac833cba7af67b7e347c"
server_key_iv: "4bac5a3ad344041ea9b7a9ea2b13117d"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.240736 19969 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:03:52.241811 19984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.242256 19969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:52.242403 19969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "a402d3f2b5364a93afae675d51ee78b5"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "8a08cc7aafa5ac833cba7af67b7e347c"
server_key_iv: "4bac5a3ad344041ea9b7a9ea2b13117d"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.242501 19969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:52.250171 19969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:52.252390 19969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:52.252562 19969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:52.257313 19969 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:43583
I20260619 12:03:52.257378 20036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:43583 every 8 connection(s)
I20260619 12:03:52.257778 19969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:52.258947 20037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:52.259485 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 19969
I20260619 12:03:52.259617 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:52.259829 815 external_mini_cluster.cc:1428] Setting key a022e650858f86a9169050dc51541e56
I20260619 12:03:52.262724 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5: Bootstrap starting.
I20260619 12:03:52.263419 20037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:52.263767 20037 log.cc:824] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:52.264531 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5: No bootstrap required, opened a new log
I20260619 12:03:52.267146 20037 raft_consensus.cc:348] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a402d3f2b5364a93afae675d51ee78b5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43583 } }
I20260619 12:03:52.267418 20037 raft_consensus.cc:374] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:52.267457 20037 raft_consensus.cc:729] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a402d3f2b5364a93afae675d51ee78b5, State: Initialized, Role: FOLLOWER
I20260619 12:03:52.267760 20037 consensus_queue.cc:260] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a402d3f2b5364a93afae675d51ee78b5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43583 } }
I20260619 12:03:52.267897 20037 raft_consensus.cc:388] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:52.267936 20037 raft_consensus.cc:482] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:52.267983 20037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:52.268863 20037 raft_consensus.cc:504] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a402d3f2b5364a93afae675d51ee78b5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43583 } }
I20260619 12:03:52.269006 20037 leader_election.cc:302] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a402d3f2b5364a93afae675d51ee78b5; no voters:
I20260619 12:03:52.269279 20037 leader_election.cc:288] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:52.269313 20042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:52.269475 20042 raft_consensus.cc:686] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [term 1 LEADER]: Becoming Leader. State: Replica: a402d3f2b5364a93afae675d51ee78b5, State: Running, Role: LEADER
I20260619 12:03:52.269577 20042 consensus_queue.cc:237] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a402d3f2b5364a93afae675d51ee78b5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43583 } }
I20260619 12:03:52.269618 20037 sys_catalog.cc:564] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:52.269942 20042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a402d3f2b5364a93afae675d51ee78b5. Latest consensus state: current_term: 1 leader_uuid: "a402d3f2b5364a93afae675d51ee78b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a402d3f2b5364a93afae675d51ee78b5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43583 } } }
I20260619 12:03:52.270005 20042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:52.270196 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a402d3f2b5364a93afae675d51ee78b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a402d3f2b5364a93afae675d51ee78b5" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 43583 } } }
I20260619 12:03:52.270287 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:52.270588 20046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:52.271418 20046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:52.273524 20046 catalog_manager.cc:1347] Generated new cluster ID: bc4be76f857f42da8c2c4bf93797d8ec
I20260619 12:03:52.273578 20046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:52.273744 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.260524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.203.254:43583)
0619 12:03:52.260745 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.260756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.260931 (+ 175us) server_negotiation.cc:407] Connection header received
0619 12:03:52.261562 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.261568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.261624 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.261753 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:52.262711 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.263810 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.264678 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.265128 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.265261 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.268839 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:52.268844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:52.268844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:52.268859 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:52.271730 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.272248 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.272251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.272251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.272356 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.272557 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.272558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.272558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.272910 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:52.273087 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.273346 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.273525 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
I20260619 12:03:52.283535 20046 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:52.284251 20046 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:52.290575 20046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5: Generated new TSK 0
I20260619 12:03:52.290828 20046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.203.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
Entry for principal kudu/127.0.203.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab.
I20260619 12:03:52.328368 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43583
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:52.415585 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:52.415849 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:52.415892 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:52.417219 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:52.417294 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:52.417342 20063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:52.418941 20063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.193.keytab
--principal=kudu/127.0.203.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:52.419239 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:52.419498 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:52.425573 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:52.425614 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:52.425868 20063 server_base.cc:1034] running on GCE node
W20260619 12:03:52.425573 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:52.426235 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:52.426556 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:52.427757 20063 hybrid_clock.cc:648] HybridClock initialized: now 1781870632427729 us; error 51 us; skew 500 ppm
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:52.430774 20063 init.cc:377] Logged in from keytab as kudu/127.0.203.193@KRBTEST.COM (short username kudu)
I20260619 12:03:52.431288 20063 webserver.cc:460] Webserver started at http://127.0.203.193:35431/ using document root <none> and password file <none>
I20260619 12:03:52.431550 20063 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:52.431620 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:52.431775 20063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:52.432977 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "5c67d55a4da8469b8ff4a89d71759db4"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "0595f57a6a0d0f3e77b6ec03202987a6"
server_key_iv: "7099b4d41a3d5bb0f4312c5009f4a583"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.433337 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "5c67d55a4da8469b8ff4a89d71759db4"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "0595f57a6a0d0f3e77b6ec03202987a6"
server_key_iv: "7099b4d41a3d5bb0f4312c5009f4a583"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.434748 20063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:52.435621 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.435839 20063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:52.435964 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "5c67d55a4da8469b8ff4a89d71759db4"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "0595f57a6a0d0f3e77b6ec03202987a6"
server_key_iv: "7099b4d41a3d5bb0f4312c5009f4a583"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.436061 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:52.456157 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:52.458642 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:52.458813 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:52.459069 20063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:52.459442 20063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:52.459499 20063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.459537 20063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:52.459563 20063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.466817 20063 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:37439
I20260619 12:03:52.466863 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:37439 every 8 connection(s)
I20260619 12:03:52.467208 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.193@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:52.475831 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 20063
I20260619 12:03:52.476018 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:03:52.476248 815 external_mini_cluster.cc:1428] Setting key 2fbfdf50402725145d9cc6290a03ad8c
I20260619 12:03:52.479591 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.468183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:55383 (local address 127.0.203.254:43583)
0619 12:03:52.468270 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.468274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.468604 (+ 330us) server_negotiation.cc:407] Connection header received
0619 12:03:52.469716 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.469718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.469732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.469872 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:52.471041 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.471619 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.472758 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.472994 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.473061 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.476007 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:52.476010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:52.476011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:52.476022 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:52.477826 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.478429 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.478430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.478430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.478490 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.478829 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.478830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.478831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.478960 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:52.479040 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.479376 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.479483 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:03:52.479836 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.468173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:43583 (local address 127.0.203.193:55383)
0619 12:03:52.468419 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.468466 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.469465 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.470099 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.470105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.470442 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:52.470911 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.470922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.471723 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.471724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.472612 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.472615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.473113 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.473115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.473268 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.473917 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:52.473927 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:52.475782 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:52.477932 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:52.477943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:52.477945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:52.478335 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:52.478603 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:52.478604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:52.478605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:52.478743 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:52.479044 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:52.479051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:52.479228 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.479354 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.479557 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":153,"threads_started":1}
I20260619 12:03:52.480438 20192 heartbeater.cc:347] Connected to a master server at 127.0.203.254:43583
I20260619 12:03:52.480535 20192 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:52.480840 20192 heartbeater.cc:511] Master 127.0.203.254:43583 requested a full tablet report, sending...
I20260619 12:03:52.481527 20001 ts_manager.cc:194] Registered new tserver with Master: 5c67d55a4da8469b8ff4a89d71759db4 (127.0.203.193:37439)
I20260619 12:03:52.482201 20001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.193@KRBTEST.COM'} at 127.0.203.193:55383
WARNING: no policy specified for kudu/127.0.203.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
Entry for principal kudu/127.0.203.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab.
I20260619 12:03:52.504808 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43583
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:52.588472 20197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:52.588915 20197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:52.588999 20197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:52.590286 20197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:52.590389 20197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:52.590447 20197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:52.592136 20197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.194.keytab
--principal=kudu/127.0.203.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:52.592445 20197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:52.592733 20197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:03:52.598645 20197 server_base.cc:1034] running on GCE node
W20260619 12:03:52.598623 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:52.598626 20205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:52.598626 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:52.599208 20197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:52.599505 20197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:52.600729 20197 hybrid_clock.cc:648] HybridClock initialized: now 1781870632600725 us; error 52 us; skew 500 ppm
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:52.603454 20197 init.cc:377] Logged in from keytab as kudu/127.0.203.194@KRBTEST.COM (short username kudu)
I20260619 12:03:52.604019 20197 webserver.cc:460] Webserver started at http://127.0.203.194:43397/ using document root <none> and password file <none>
I20260619 12:03:52.604264 20197 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:52.604333 20197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:52.604470 20197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:52.605679 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "14f830c834c64420a3c1de8dbb8f9152"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "580d614b4f588f1c27d6153e996b3fc8"
server_key_iv: "d34352c093e9c77aafcd30c3ce261f74"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.606040 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "14f830c834c64420a3c1de8dbb8f9152"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "580d614b4f588f1c27d6153e996b3fc8"
server_key_iv: "d34352c093e9c77aafcd30c3ce261f74"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.607489 20197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:52.608426 20212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.608777 20197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:52.608886 20197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "14f830c834c64420a3c1de8dbb8f9152"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "580d614b4f588f1c27d6153e996b3fc8"
server_key_iv: "d34352c093e9c77aafcd30c3ce261f74"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.608979 20197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:52.627015 20197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:52.629616 20197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:52.629817 20197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:52.630088 20197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:52.630448 20197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:52.630507 20197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.630543 20197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:52.630575 20197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.636930 20197 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:42877
I20260619 12:03:52.637115 20325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:42877 every 8 connection(s)
I20260619 12:03:52.637387 20197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:03:52.641317 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 20197
I20260619 12:03:52.641494 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:03:52.641714 815 external_mini_cluster.cc:1428] Setting key 72274b616572a5360dfc3f14b34115e2
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.194@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:52.649564 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.638261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:55927 (local address 127.0.203.254:43583)
0619 12:03:52.638352 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.638354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.638575 (+ 221us) server_negotiation.cc:407] Connection header received
0619 12:03:52.639814 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.639815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.639826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.639890 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:52.640798 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.641344 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.642920 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.643130 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.643221 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.646470 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:52.646472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:52.646472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:52.646481 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:52.648002 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.648496 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.648497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.648497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.648559 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.648861 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.648862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.648862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.648991 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:52.649067 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.649372 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.649457 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:03:52.649808 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.638224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:43583 (local address 127.0.203.194:55927)
0619 12:03:52.638399 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.638442 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.639654 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.639978 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.639983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.640390 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:52.640690 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.640697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.641524 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.641525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.642759 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.642763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.643243 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.643244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.643427 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.644103 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:52.644117 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:52.646351 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:52.648115 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:52.648127 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:52.648130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:52.648415 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:52.648629 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:52.648630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:52.648631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:52.648792 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:52.649071 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:52.649078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:52.649232 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.649338 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.649539 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":108,"threads_started":1}
I20260619 12:03:52.650419 20326 heartbeater.cc:347] Connected to a master server at 127.0.203.254:43583
I20260619 12:03:52.650511 20326 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:52.650730 20326 heartbeater.cc:511] Master 127.0.203.254:43583 requested a full tablet report, sending...
I20260619 12:03:52.651317 20001 ts_manager.cc:194] Registered new tserver with Master: 14f830c834c64420a3c1de8dbb8f9152 (127.0.203.194:42877)
I20260619 12:03:52.652052 20001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.194@KRBTEST.COM'} at 127.0.203.194:55927
WARNING: no policy specified for kudu/127.0.203.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
Entry for principal kudu/127.0.203.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab.
I20260619 12:03:52.676229 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43583
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:52.770987 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:52.771203 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:52.771229 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:52.772441 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:52.772495 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:52.772522 20331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:52.774044 20331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/krb5kdc/kudu_127.0.203.195.keytab
--principal=kudu/127.0.203.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:43583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:52.774395 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:52.774681 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:52.780716 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:52.780716 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:52.780789 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:52.780882 20331 server_base.cc:1034] running on GCE node
I20260619 12:03:52.781414 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:52.781734 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:52.782928 20331 hybrid_clock.cc:648] HybridClock initialized: now 1781870632782907 us; error 41 us; skew 500 ppm
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:52.786146 20331 init.cc:377] Logged in from keytab as kudu/127.0.203.195@KRBTEST.COM (short username kudu)
I20260619 12:03:52.786690 20331 webserver.cc:460] Webserver started at http://127.0.203.195:44863/ using document root <none> and password file <none>
I20260619 12:03:52.786970 20331 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:52.787040 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:52.787196 20331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:52.788436 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "f07e01eff05a4253b68c3f0cd5933190"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "518d27e81de6da00df5a0c52b4d02ba6"
server_key_iv: "8709f551e2884149638f7061b3073afc"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.788899 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "f07e01eff05a4253b68c3f0cd5933190"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "518d27e81de6da00df5a0c52b4d02ba6"
server_key_iv: "8709f551e2884149638f7061b3073afc"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.790475 20331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:52.791494 20346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.791733 20331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:52.791836 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "f07e01eff05a4253b68c3f0cd5933190"
format_stamp: "Formatted at 2026-06-19 12:03:52 on dist-test-slave-fbq8"
server_key: "518d27e81de6da00df5a0c52b4d02ba6"
server_key_iv: "8709f551e2884149638f7061b3073afc"
server_key_version: "encryptionkey@0"
I20260619 12:03:52.791924 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:52.806824 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:52.809154 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:52.809324 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:52.809612 20331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:52.809955 20331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:52.809995 20331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.810056 20331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:52.810089 20331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:52.816591 20331 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:35133
I20260619 12:03:52.816776 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:35133 every 8 connection(s)
I20260619 12:03:52.817101 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:03:52.822604 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 20331
I20260619 12:03:52.822727 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:03:52.823064 815 external_mini_cluster.cc:1428] Setting key 7ba70dc237ccf02af57026789efa018c
Jun 19 12:03:52 dist-test-slave-fbq8 krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.195@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:03:52.828006 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.818008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:49945 (local address 127.0.203.254:43583)
0619 12:03:52.818085 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.818087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.818358 (+ 271us) server_negotiation.cc:407] Connection header received
0619 12:03:52.819458 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.819459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.819473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.819530 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:52.820252 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.820887 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.821917 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.822128 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.822224 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.825065 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:52.825066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:52.825067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:52.825075 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:52.826404 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.826867 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.826868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.826869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.826932 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.827186 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.827187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.827188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.827312 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:52.827388 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.827797 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.827890 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:03:52.828229 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.817985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:43583 (local address 127.0.203.195:49945)
0619 12:03:52.818186 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.818241 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.819325 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.819611 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.819615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.819920 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:52.820141 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.820145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.820974 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.820975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.821826 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.821828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.822242 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.822243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.822370 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.823192 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:03:52.823205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:52.824967 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:52.826517 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:52.826524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:52.826527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:52.826789 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:52.826998 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:03:52.826999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:03:52.826999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:03:52.827118 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:03:52.827389 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:52.827396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:52.827542 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.827762 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.827961 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260619 12:03:52.828984 20460 heartbeater.cc:347] Connected to a master server at 127.0.203.254:43583
I20260619 12:03:52.829172 20460 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:52.829469 20460 heartbeater.cc:511] Master 127.0.203.254:43583 requested a full tablet report, sending...
I20260619 12:03:52.830062 20001 ts_manager.cc:194] Registered new tserver with Master: f07e01eff05a4253b68c3f0cd5933190 (127.0.203.195:35133)
I20260619 12:03:52.830523 20001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.203.195@KRBTEST.COM'} at 127.0.203.195:49945
I20260619 12:03:52.835532 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:03:52.844112 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.837062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.0.203.254:43583)
0619 12:03:52.837197 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.837200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.837495 (+ 295us) server_negotiation.cc:407] Connection header received
0619 12:03:52.837600 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.837604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.837615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.837700 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:52.838338 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.838952 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.839779 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.839980 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.840041 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.841414 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:52.841416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:52.841416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:52.841422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:52.842862 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.843252 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.843253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.843253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.843302 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:52.843509 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:52.843510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:52.843510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:52.843665 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:52.843750 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.843948 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.844014 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 12:03:52.846220 20001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:03:52.848415 20001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:03:52.856719 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.851672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:37439 (local address 127.0.0.1:38300)
0619 12:03:52.852472 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.852486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.852556 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.853238 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.853239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.853241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:52.853290 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.853291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.854466 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.854467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.855338 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.855340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.856339 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.856340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.856428 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.856434 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.856463 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.856559 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":7,"thread_start_us":238,"threads_started":1}
I20260619 12:03:52.856803 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.851815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.203.195:35133)
0619 12:03:52.852053 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.852059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.852173 (+ 114us) server_negotiation.cc:407] Connection header received
0619 12:03:52.852343 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.852346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.852364 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.852435 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:52.852770 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.854020 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.855599 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.856161 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.856257 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.856320 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.856605 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.856696 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":122,"threads_started":1}
I20260619 12:03:52.857188 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.852353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.0.203.193:37439)
0619 12:03:52.853074 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.853078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.853087 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:03:52.853130 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.853132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.853146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.853235 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:52.853370 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.854374 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.855438 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.856228 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.856961 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.857031 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.857058 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.857103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":86,"threads_started":1}
I20260619 12:03:52.857336 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.851672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:35133 (local address 127.0.0.1:43668)
0619 12:03:52.852049 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.852094 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.852196 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.852540 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.852543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.852546 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:52.852631 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.852633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.854149 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.854150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.855475 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.855478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.856259 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.856261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.856358 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.856372 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.856472 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.857238 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":71,"thread_start_us":220,"threads_started":1}
I20260619 12:03:52.857959 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.852102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:42877 (local address 127.0.0.1:34596)
0619 12:03:52.852533 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.852543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.852593 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.852938 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.852939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.852942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:52.852993 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.852994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.854083 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.854084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.856556 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.856558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.857704 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.857705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.857769 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.857773 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.857827 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.857888 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":1251,"thread_start_us":134,"threads_started":1}
I20260619 12:03:52.858152 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.852252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.0.203.194:42877)
0619 12:03:52.852712 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.852718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.852729 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:52.852789 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.852791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.852807 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.852909 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:52.853146 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.853967 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.856703 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.857521 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.857581 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.857634 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.857880 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.857961 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":114,"threads_started":1}
I20260619 12:03:52.858052 20126 tablet_service.cc:1467] Processing CreateTablet for tablet 85ec583db417414fb62bfa185b0dbcd7 (DEFAULT_TABLE table=test-table [id=019051b8d2fc44c29728060af122d1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:52.858158 20394 tablet_service.cc:1467] Processing CreateTablet for tablet 85ec583db417414fb62bfa185b0dbcd7 (DEFAULT_TABLE table=test-table [id=019051b8d2fc44c29728060af122d1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:52.858513 20126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ec583db417414fb62bfa185b0dbcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:52.858597 20394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ec583db417414fb62bfa185b0dbcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:52.858896 20260 tablet_service.cc:1467] Processing CreateTablet for tablet 85ec583db417414fb62bfa185b0dbcd7 (DEFAULT_TABLE table=test-table [id=019051b8d2fc44c29728060af122d1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:52.859189 20260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ec583db417414fb62bfa185b0dbcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:52.861899 20477 tablet_bootstrap.cc:492] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4: Bootstrap starting.
I20260619 12:03:52.862401 20479 tablet_bootstrap.cc:492] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Bootstrap starting.
I20260619 12:03:52.862799 20477 tablet_bootstrap.cc:654] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:52.863011 20479 tablet_bootstrap.cc:654] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:52.863422 20477 log.cc:824] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:52.863587 20479 log.cc:824] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:52.863824 20478 tablet_bootstrap.cc:492] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Bootstrap starting.
I20260619 12:03:52.864281 20477 tablet_bootstrap.cc:492] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4: No bootstrap required, opened a new log
I20260619 12:03:52.864281 20479 tablet_bootstrap.cc:492] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: No bootstrap required, opened a new log
I20260619 12:03:52.864357 20479 ts_tablet_manager.cc:1397] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:03:52.864357 20477 ts_tablet_manager.cc:1397] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:03:52.864389 20478 tablet_bootstrap.cc:654] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:52.864938 20478 log.cc:824] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:52.865912 20478 tablet_bootstrap.cc:492] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: No bootstrap required, opened a new log
I20260619 12:03:52.866014 20478 ts_tablet_manager.cc:1397] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 12:03:52.866603 20477 raft_consensus.cc:348] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.866812 20477 raft_consensus.cc:374] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:52.866735 20479 raft_consensus.cc:348] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.866852 20477 raft_consensus.cc:729] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c67d55a4da8469b8ff4a89d71759db4, State: Initialized, Role: FOLLOWER
I20260619 12:03:52.866868 20479 raft_consensus.cc:374] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:52.866900 20479 raft_consensus.cc:729] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f830c834c64420a3c1de8dbb8f9152, State: Initialized, Role: FOLLOWER
I20260619 12:03:52.867008 20477 consensus_queue.cc:260] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.867007 20479 consensus_queue.cc:260] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.867307 20477 ts_tablet_manager.cc:1428] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:52.867307 20479 ts_tablet_manager.cc:1428] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 12:03:52.867426 20192 heartbeater.cc:503] Master 127.0.203.254:43583 was elected leader, sending a full tablet report...
I20260619 12:03:52.867462 20326 heartbeater.cc:503] Master 127.0.203.254:43583 was elected leader, sending a full tablet report...
I20260619 12:03:52.868402 20478 raft_consensus.cc:348] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.868580 20478 raft_consensus.cc:374] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:52.868604 20478 raft_consensus.cc:729] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07e01eff05a4253b68c3f0cd5933190, State: Initialized, Role: FOLLOWER
I20260619 12:03:52.868816 20478 consensus_queue.cc:260] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.869179 20478 ts_tablet_manager.cc:1428] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 12:03:52.869385 20460 heartbeater.cc:503] Master 127.0.203.254:43583 was elected leader, sending a full tablet report...
W20260619 12:03:52.888051 20327 tablet.cc:2369] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:03:52.959010 20484 raft_consensus.cc:482] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:52.959190 20484 raft_consensus.cc:504] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.959676 20484 leader_election.cc:288] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f07e01eff05a4253b68c3f0cd5933190 (127.0.203.195:35133), 14f830c834c64420a3c1de8dbb8f9152 (127.0.203.194:42877)
I20260619 12:03:52.964491 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.960077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:40221 (local address 127.0.203.195:35133)
0619 12:03:52.960174 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.960177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.960194 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:03:52.960243 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.960244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.960258 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.960350 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:52.960559 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.961459 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.963334 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.964053 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.964130 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.964138 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.964357 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.964400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:03:52.964491 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.960184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:46819 (local address 127.0.203.194:42877)
0619 12:03:52.960251 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.960254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.960458 (+ 204us) server_negotiation.cc:407] Connection header received
0619 12:03:52.960485 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.960486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.960495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.960572 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:52.960821 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.961587 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.963335 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.964013 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.964117 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.964129 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.964362 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.964419 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 12:03:52.964535 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.959918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:35133 (local address 127.0.203.193:40221)
0619 12:03:52.960045 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.960064 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.960175 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.960371 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.960372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.960374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:52.960471 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.960474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.961625 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.961669 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.963214 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.963217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.964165 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.964166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.964268 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.964271 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.964338 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.964404 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":95}
I20260619 12:03:52.964543 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.960077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:42877 (local address 127.0.203.193:46819)
0619 12:03:52.960283 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.960293 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:52.960385 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:52.960588 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:52.960589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:52.960590 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:52.960724 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.960726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.961711 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.961712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.963214 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:52.963217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.964165 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.964166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:52.964256 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.964262 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:03:52.964338 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 12:03:52.964403 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":59,"thread_start_us":104,"threads_started":1}
I20260619 12:03:52.965071 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec583db417414fb62bfa185b0dbcd7" candidate_uuid: "5c67d55a4da8469b8ff4a89d71759db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07e01eff05a4253b68c3f0cd5933190" is_pre_election: true
I20260619 12:03:52.965274 20414 raft_consensus.cc:2454] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c67d55a4da8469b8ff4a89d71759db4 in term 0.
I20260619 12:03:52.965245 20280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec583db417414fb62bfa185b0dbcd7" candidate_uuid: "5c67d55a4da8469b8ff4a89d71759db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f830c834c64420a3c1de8dbb8f9152" is_pre_election: true
I20260619 12:03:52.965376 20280 raft_consensus.cc:2454] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c67d55a4da8469b8ff4a89d71759db4 in term 0.
I20260619 12:03:52.965610 20081 leader_election.cc:302] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c67d55a4da8469b8ff4a89d71759db4, f07e01eff05a4253b68c3f0cd5933190; no voters:
I20260619 12:03:52.965817 20484 raft_consensus.cc:2781] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:03:52.965919 20484 raft_consensus.cc:482] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:03:52.965938 20484 raft_consensus.cc:3037] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:52.966843 20484 raft_consensus.cc:504] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.966984 20484 leader_election.cc:288] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [CANDIDATE]: Term 1 election: Requested vote from peers f07e01eff05a4253b68c3f0cd5933190 (127.0.203.195:35133), 14f830c834c64420a3c1de8dbb8f9152 (127.0.203.194:42877)
I20260619 12:03:52.967217 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec583db417414fb62bfa185b0dbcd7" candidate_uuid: "5c67d55a4da8469b8ff4a89d71759db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f07e01eff05a4253b68c3f0cd5933190"
I20260619 12:03:52.967244 20280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ec583db417414fb62bfa185b0dbcd7" candidate_uuid: "5c67d55a4da8469b8ff4a89d71759db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f830c834c64420a3c1de8dbb8f9152"
I20260619 12:03:52.967321 20414 raft_consensus.cc:3037] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:52.967350 20280 raft_consensus.cc:3037] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:52.968168 20414 raft_consensus.cc:2454] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c67d55a4da8469b8ff4a89d71759db4 in term 1.
I20260619 12:03:52.968192 20280 raft_consensus.cc:2454] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c67d55a4da8469b8ff4a89d71759db4 in term 1.
I20260619 12:03:52.968345 20081 leader_election.cc:302] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c67d55a4da8469b8ff4a89d71759db4, f07e01eff05a4253b68c3f0cd5933190; no voters:
I20260619 12:03:52.968451 20484 raft_consensus.cc:2781] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:52.968675 20484 raft_consensus.cc:686] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [term 1 LEADER]: Becoming Leader. State: Replica: 5c67d55a4da8469b8ff4a89d71759db4, State: Running, Role: LEADER
W20260619 12:03:52.968870 20193 tablet.cc:2369] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:03:52.968838 20484 consensus_queue.cc:237] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } }
I20260619 12:03:52.970113 20000 catalog_manager.cc:5495] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c67d55a4da8469b8ff4a89d71759db4 (127.0.203.193). New cstate: current_term: 1 leader_uuid: "5c67d55a4da8469b8ff4a89d71759db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c67d55a4da8469b8ff4a89d71759db4" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 } health_report { overall_health: UNKNOWN } } }
I20260619 12:03:52.976290 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.973740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.203.193:37439)
0619 12:03:52.973783 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:52.973786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:52.973796 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:03:52.973865 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:52.973866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:52.973883 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:52.973955 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:52.974153 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.974643 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.975440 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:52.975618 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:52.975671 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:52.975812 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:52.975919 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:03:52.975987 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:52.976155 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:03:52.976191 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 12:03:52.978411 20280 raft_consensus.cc:1261] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 1 FOLLOWER]: Refusing update from remote peer 5c67d55a4da8469b8ff4a89d71759db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:52.978411 20414 raft_consensus.cc:1261] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 1 FOLLOWER]: Refusing update from remote peer 5c67d55a4da8469b8ff4a89d71759db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:52.978773 20487 consensus_queue.cc:1035] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f830c834c64420a3c1de8dbb8f9152" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:52.978924 20484 consensus_queue.cc:1035] T 85ec583db417414fb62bfa185b0dbcd7 P 5c67d55a4da8469b8ff4a89d71759db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f07e01eff05a4253b68c3f0cd5933190" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:52.983386 20492 mvcc.cc:205] Tried to move back new op lower bound from 7298542112676667392 to 7298542112641552384. Current Snapshot: MvccSnapshot[applied={T|T < 7298542112676667392}]
I20260619 12:03:52.984265 20491 mvcc.cc:205] Tried to move back new op lower bound from 7298542112676667392 to 7298542112641552384. Current Snapshot: MvccSnapshot[applied={T|T < 7298542112676667392}]
I20260619 12:03:52.985116 20000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:52.985239 20000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:52.986191 20000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a402d3f2b5364a93afae675d51ee78b5: Sending DeleteTablet for 3 replicas of tablet 85ec583db417414fb62bfa185b0dbcd7
I20260619 12:03:52.986588 20394 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ec583db417414fb62bfa185b0dbcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:52 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:43668
I20260619 12:03:52.986582 20126 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ec583db417414fb62bfa185b0dbcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:52 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:38300
I20260619 12:03:52.986882 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 20063
I20260619 12:03:52.986901 20260 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ec583db417414fb62bfa185b0dbcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:52 UTC) from {username='kudu', principal='kudu/127.0.203.254@KRBTEST.COM'} at 127.0.0.1:34596
I20260619 12:03:52.987586 20501 tablet_replica.cc:330] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: stopping tablet replica
I20260619 12:03:52.987677 20501 raft_consensus.cc:2229] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:52.987762 20501 raft_consensus.cc:2258] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:52.988221 20501 ts_tablet_manager.cc:1905] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:52.989482 20502 tablet_replica.cc:330] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: stopping tablet replica
I20260619 12:03:52.989598 20502 raft_consensus.cc:2229] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:52.989727 20502 raft_consensus.cc:2258] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:52.990267 20502 ts_tablet_manager.cc:1905] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:52.990307 20501 ts_tablet_manager.cc:1918] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:52.990368 20501 log.cc:1196] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/85ec583db417414fb62bfa185b0dbcd7
I20260619 12:03:52.990650 20501 ts_tablet_manager.cc:1939] T 85ec583db417414fb62bfa185b0dbcd7 P f07e01eff05a4253b68c3f0cd5933190: Deleting consensus metadata
I20260619 12:03:52.991245 19987 catalog_manager.cc:4841] TS f07e01eff05a4253b68c3f0cd5933190 (127.0.203.195:35133): tablet 85ec583db417414fb62bfa185b0dbcd7 (table test-table [id=019051b8d2fc44c29728060af122d1cd]) successfully deleted
I20260619 12:03:52.992272 20502 ts_tablet_manager.cc:1918] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:52.992334 20502 log.cc:1196] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/85ec583db417414fb62bfa185b0dbcd7
I20260619 12:03:52.992550 20502 ts_tablet_manager.cc:1939] T 85ec583db417414fb62bfa185b0dbcd7 P 14f830c834c64420a3c1de8dbb8f9152: Deleting consensus metadata
I20260619 12:03:52.993080 19987 catalog_manager.cc:4841] TS 14f830c834c64420a3c1de8dbb8f9152 (127.0.203.194:42877): tablet 85ec583db417414fb62bfa185b0dbcd7 (table test-table [id=019051b8d2fc44c29728060af122d1cd]) successfully deleted
W20260619 12:03:52.993721 19988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:37439 (error 108)
I20260619 12:03:52.994431 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:52.994277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:37439 (local address 127.0.0.1:38326)
0619 12:03:52.994346 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:52.994390 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 12:03:52.994527 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 20197
W20260619 12:03:52.994578 19988 catalog_manager.cc:4584] TS 5c67d55a4da8469b8ff4a89d71759db4 (127.0.203.193:37439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85ec583db417414fb62bfa185b0dbcd7: Network error: Client connection negotiation failed: client connection to 127.0.203.193:37439: connect: Connection refused (error 111)
I20260619 12:03:53.001008 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 20331
I20260619 12:03:53.007513 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 19969
2026-06-19T12:03:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2091 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T12:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:53Z Disabled control of system clock
I20260619 12:03:53.038575 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40027
--webserver_interface=127.0.203.254
--webserver_port=0
--builtin_ntp_servers=127.0.203.212:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:40027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:03:53.123786 20507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:53.124046 20507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:53.124068 20507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:53.125386 20507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:53.125444 20507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:53.125458 20507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:53.125471 20507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:53.125484 20507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:53.126941 20507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:40027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:53.127239 20507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:53.127591 20507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:53.132987 20512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:53.133033 20515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:53.133323 20507 server_base.cc:1034] running on GCE node
W20260619 12:03:53.132987 20513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:53.133664 20507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:53.133989 20507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:53.135169 20507 hybrid_clock.cc:648] HybridClock initialized: now 1781870633135147 us; error 61 us; skew 500 ppm
I20260619 12:03:53.136699 20507 webserver.cc:460] Webserver started at http://127.0.203.254:45175/ using document root <none> and password file <none>
I20260619 12:03:53.136981 20507 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:53.137050 20507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:53.137184 20507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:53.138289 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "038b6e52e20841239eb5097cec8b06df"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "4e48089c5246bb5944c791f3158d7e78"
server_key_iv: "8214b11d1861da98d9e43176dc6ccd51"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.138670 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "038b6e52e20841239eb5097cec8b06df"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "4e48089c5246bb5944c791f3158d7e78"
server_key_iv: "8214b11d1861da98d9e43176dc6ccd51"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.140070 20507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:03:53.141022 20521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.141297 20507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:53.141422 20507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "038b6e52e20841239eb5097cec8b06df"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "4e48089c5246bb5944c791f3158d7e78"
server_key_iv: "8214b11d1861da98d9e43176dc6ccd51"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.141541 20507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:53.148904 20507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:53.149240 20507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:53.149389 20507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:53.153967 20507 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:40027
I20260619 12:03:53.154036 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:40027 every 8 connection(s)
I20260619 12:03:53.154410 20507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:53.155550 20574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:53.158644 20574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df: Bootstrap starting.
I20260619 12:03:53.159389 20574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:53.160019 20574 log.cc:824] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:53.161088 20574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df: No bootstrap required, opened a new log
I20260619 12:03:53.163116 20574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038b6e52e20841239eb5097cec8b06df" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40027 } }
I20260619 12:03:53.163306 20574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:53.163331 20574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038b6e52e20841239eb5097cec8b06df, State: Initialized, Role: FOLLOWER
I20260619 12:03:53.163571 20574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038b6e52e20841239eb5097cec8b06df" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40027 } }
I20260619 12:03:53.163671 20574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:53.163708 20574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:53.163707 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 20507
I20260619 12:03:53.163769 20574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:53.163856 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:53.164079 815 external_mini_cluster.cc:1428] Setting key 646222b6786c91736eedbbd93fa75452
I20260619 12:03:53.164592 20574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038b6e52e20841239eb5097cec8b06df" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40027 } }
I20260619 12:03:53.164748 20574 leader_election.cc:302] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 038b6e52e20841239eb5097cec8b06df; no voters:
I20260619 12:03:53.164939 20574 leader_election.cc:288] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:53.165031 20579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:53.165246 20579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [term 1 LEADER]: Becoming Leader. State: Replica: 038b6e52e20841239eb5097cec8b06df, State: Running, Role: LEADER
I20260619 12:03:53.165274 20574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:53.165361 20579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038b6e52e20841239eb5097cec8b06df" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40027 } }
I20260619 12:03:53.165759 20579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038b6e52e20841239eb5097cec8b06df. Latest consensus state: current_term: 1 leader_uuid: "038b6e52e20841239eb5097cec8b06df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038b6e52e20841239eb5097cec8b06df" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40027 } } }
I20260619 12:03:53.165854 20579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:53.165992 20580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "038b6e52e20841239eb5097cec8b06df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "038b6e52e20841239eb5097cec8b06df" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40027 } } }
I20260619 12:03:53.166473 20580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:53.167008 20586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:53.167672 20586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:53.169570 20586 catalog_manager.cc:1347] Generated new cluster ID: ab9425c482ea43228bafc3c5277b957c
I20260619 12:03:53.169625 20586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:53.171754 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.164686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.0.203.254:40027)
0619 12:03:53.164991 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.165004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.165057 (+ 53us) server_negotiation.cc:407] Connection header received
0619 12:03:53.165226 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.165234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.165333 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.165855 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:53.165979 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.168029 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.169223 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.169838 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.169981 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.170874 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:53.170878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:53.170879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:53.171141 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:53.171191 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:53.171207 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:53.171289 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.171368 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.171462 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":135,"threads_started":1}
I20260619 12:03:53.182361 20586 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:53.182897 20586 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:53.191318 20586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df: Generated new TSK 0
I20260619 12:03:53.191694 20586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:03:53.195348 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.193:0
--local_ip_for_outbound_sockets=127.0.203.193
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40027
--builtin_ntp_servers=127.0.203.212:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:03:53.301999 20598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:53.302345 20598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:53.302383 20598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:53.303792 20598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:53.303861 20598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:53.303908 20598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.193
I20260619 12:03:53.305383 20598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.203.193
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.193
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:53.305666 20598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:53.306002 20598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:53.311605 20604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:53.311728 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:53.311890 20598 server_base.cc:1034] running on GCE node
W20260619 12:03:53.311934 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:53.312228 20598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:53.312537 20598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:53.313715 20598 hybrid_clock.cc:648] HybridClock initialized: now 1781870633313696 us; error 44 us; skew 500 ppm
I20260619 12:03:53.315183 20598 webserver.cc:460] Webserver started at http://127.0.203.193:37009/ using document root <none> and password file <none>
I20260619 12:03:53.315470 20598 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:53.315547 20598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:53.315707 20598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:53.317003 20598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data/instance:
uuid: "116f7f5a0c994ae882922f8667d081ac"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "5e02320b4f3f2d5b56d8d4ec74a1ba25"
server_key_iv: "7e4dbc3f439dee4fdc96fc70be4a8149"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.317377 20598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/wal/instance:
uuid: "116f7f5a0c994ae882922f8667d081ac"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "5e02320b4f3f2d5b56d8d4ec74a1ba25"
server_key_iv: "7e4dbc3f439dee4fdc96fc70be4a8149"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.318755 20598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:53.319522 20612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.319808 20598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:53.319967 20598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/wal
uuid: "116f7f5a0c994ae882922f8667d081ac"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "5e02320b4f3f2d5b56d8d4ec74a1ba25"
server_key_iv: "7e4dbc3f439dee4fdc96fc70be4a8149"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.320086 20598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:53.329947 20598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:53.330257 20598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:53.330370 20598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:53.330585 20598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:53.330895 20598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:53.330931 20598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.330955 20598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:53.330967 20598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.337708 20598 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.193:45547
I20260619 12:03:53.337831 20725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.193:45547 every 8 connection(s)
I20260619 12:03:53.338189 20598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/data/info.pb
I20260619 12:03:53.341917 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 20598
I20260619 12:03:53.342011 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-0/wal/instance
I20260619 12:03:53.342257 815 external_mini_cluster.cc:1428] Setting key 74281821651507717cf2fec65e8b900f
I20260619 12:03:53.343539 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.194:0
--local_ip_for_outbound_sockets=127.0.203.194
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40027
--builtin_ntp_servers=127.0.203.212:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:03:53.345432 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.339138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:32841 (local address 127.0.203.254:40027)
0619 12:03:53.339222 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.339225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.339569 (+ 344us) server_negotiation.cc:407] Connection header received
0619 12:03:53.340241 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.340244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.340259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.340350 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:53.341372 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.341938 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.343440 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.343751 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.343851 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.345071 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:53.345072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:53.345073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:53.345078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:53.345094 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:53.345098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:53.345163 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.345305 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.345344 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:03:53.345713 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.339125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40027 (local address 127.0.203.193:32841)
0619 12:03:53.339391 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.339435 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.340096 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.340483 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.340489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.340539 (+ 50us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:53.341255 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.341260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.342090 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.342092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.343323 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.343327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.343880 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.343881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.344104 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.344880 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:53.344881 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:53.344915 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:53.344955 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:53.345165 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:53.345172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:53.345176 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.345305 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.345437 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":112,"threads_started":1}
I20260619 12:03:53.346433 20726 heartbeater.cc:347] Connected to a master server at 127.0.203.254:40027
I20260619 12:03:53.346560 20726 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:53.346789 20726 heartbeater.cc:511] Master 127.0.203.254:40027 requested a full tablet report, sending...
I20260619 12:03:53.347371 20538 ts_manager.cc:194] Registered new tserver with Master: 116f7f5a0c994ae882922f8667d081ac (127.0.203.193:45547)
I20260619 12:03:53.348024 20538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.193:32841
W20260619 12:03:53.438283 20729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:53.438511 20729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:53.438534 20729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:53.439694 20729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:53.439738 20729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:53.439762 20729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.194
I20260619 12:03:53.441129 20729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.203.194
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.194
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:53.441350 20729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:53.441629 20729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:53.447427 20737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:53.447463 20735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:53.447438 20734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:53.447649 20729 server_base.cc:1034] running on GCE node
I20260619 12:03:53.447877 20729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:53.448092 20729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:53.449283 20729 hybrid_clock.cc:648] HybridClock initialized: now 1781870633449266 us; error 45 us; skew 500 ppm
I20260619 12:03:53.450606 20729 webserver.cc:460] Webserver started at http://127.0.203.194:33249/ using document root <none> and password file <none>
I20260619 12:03:53.450841 20729 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:53.450891 20729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:53.450989 20729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:53.452090 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data/instance:
uuid: "93b87f5419474e2dac78e0e2687a087e"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "0c7d769bd8881570b7bbd63572e3aeb0"
server_key_iv: "e754bde14eacf2590f5faed798ad7cf9"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.452419 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal/instance:
uuid: "93b87f5419474e2dac78e0e2687a087e"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "0c7d769bd8881570b7bbd63572e3aeb0"
server_key_iv: "e754bde14eacf2590f5faed798ad7cf9"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.453989 20729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:03:53.454950 20743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.455358 20729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:53.455511 20729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal
uuid: "93b87f5419474e2dac78e0e2687a087e"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "0c7d769bd8881570b7bbd63572e3aeb0"
server_key_iv: "e754bde14eacf2590f5faed798ad7cf9"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.455626 20729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:53.470625 20729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:53.470973 20729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:53.471131 20729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:53.471405 20729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:53.471764 20729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:53.471802 20729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.471827 20729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:53.471839 20729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.479220 20729 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.194:42277
I20260619 12:03:53.479275 20856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.194:42277 every 8 connection(s)
I20260619 12:03:53.479624 20729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/data/info.pb
I20260619 12:03:53.479981 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 20729
I20260619 12:03:53.480129 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal/instance
I20260619 12:03:53.480355 815 external_mini_cluster.cc:1428] Setting key 26575cb1f2a23f5a9d91fc1f58c9849a
I20260619 12:03:53.481609 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.203.195:0
--local_ip_for_outbound_sockets=127.0.203.195
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40027
--builtin_ntp_servers=127.0.203.212:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:03:53.486827 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.480548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.194:55995 (local address 127.0.203.254:40027)
0619 12:03:53.480700 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.480703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.481403 (+ 700us) server_negotiation.cc:407] Connection header received
0619 12:03:53.482128 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.482129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.482145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.482226 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:53.483071 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.483662 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.485097 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.485335 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.485453 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.486520 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:53.486521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:53.486522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:53.486529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:53.486548 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:53.486552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:53.486610 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.486705 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.486739 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:03:53.487084 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.480551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40027 (local address 127.0.203.194:55995)
0619 12:03:53.481229 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.481288 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.482000 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.482301 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.482305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.482314 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:53.482883 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.482928 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.483765 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.483766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.484988 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.484991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.485464 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.485465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.485619 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.486336 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:53.486336 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:53.486369 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:53.486394 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:53.486611 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:53.486619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:53.486622 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.486708 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.486816 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":151,"threads_started":1}
I20260619 12:03:53.487620 20857 heartbeater.cc:347] Connected to a master server at 127.0.203.254:40027
I20260619 12:03:53.487761 20857 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:53.488022 20857 heartbeater.cc:511] Master 127.0.203.254:40027 requested a full tablet report, sending...
I20260619 12:03:53.488499 20538 ts_manager.cc:194] Registered new tserver with Master: 93b87f5419474e2dac78e0e2687a087e (127.0.203.194:42277)
I20260619 12:03:53.488926 20538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.194:55995
W20260619 12:03:53.572773 20860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:53.573010 20860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:53.573033 20860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:53.574232 20860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:53.574275 20860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:53.574302 20860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.203.195
I20260619 12:03:53.575646 20860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.203.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.203.195
--webserver_port=0
--tserver_master_addrs=127.0.203.254:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.203.195
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:53.575924 20860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:53.576267 20860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:53.582003 20866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:53.582078 20868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:53.582003 20865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:53.582278 20860 server_base.cc:1034] running on GCE node
I20260619 12:03:53.582638 20860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:53.582932 20860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:53.584124 20860 hybrid_clock.cc:648] HybridClock initialized: now 1781870633584100 us; error 46 us; skew 500 ppm
I20260619 12:03:53.585888 20860 webserver.cc:460] Webserver started at http://127.0.203.195:38109/ using document root <none> and password file <none>
I20260619 12:03:53.586174 20860 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:53.586248 20860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:53.586412 20860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:53.587674 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data/instance:
uuid: "16ee594cf0b94efb89b346e61da12478"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "b2c796aca1a1b53c197ac1349b89f584"
server_key_iv: "a9f168b5c0c53dfe2454d74ff6340b98"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.588065 20860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal/instance:
uuid: "16ee594cf0b94efb89b346e61da12478"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "b2c796aca1a1b53c197ac1349b89f584"
server_key_iv: "a9f168b5c0c53dfe2454d74ff6340b98"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.589790 20860 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 12:03:53.590790 20874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.591079 20860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:03:53.591192 20860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal
uuid: "16ee594cf0b94efb89b346e61da12478"
format_stamp: "Formatted at 2026-06-19 12:03:53 on dist-test-slave-fbq8"
server_key: "b2c796aca1a1b53c197ac1349b89f584"
server_key_iv: "a9f168b5c0c53dfe2454d74ff6340b98"
server_key_version: "encryptionkey@0"
I20260619 12:03:53.591277 20860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:53.606370 20860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:53.606724 20860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:53.606869 20860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:53.607139 20860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:03:53.607527 20860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:03:53.607580 20860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.607623 20860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:03:53.607658 20860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:53.615209 20860 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.195:38945
I20260619 12:03:53.615276 20987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.195:38945 every 8 connection(s)
I20260619 12:03:53.615661 20860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/data/info.pb
I20260619 12:03:53.617709 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 20860
I20260619 12:03:53.617821 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal/instance
I20260619 12:03:53.618016 815 external_mini_cluster.cc:1428] Setting key 98edbc868b8b9f163350eb1eb1a3dfae
I20260619 12:03:53.621949 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.616690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.195:33909 (local address 127.0.203.254:40027)
0619 12:03:53.616753 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.616756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.616974 (+ 218us) server_negotiation.cc:407] Connection header received
0619 12:03:53.617537 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.617538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.617555 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.617640 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:53.618617 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.619336 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.620333 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.620529 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.620587 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.621569 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:53.621570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:53.621571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:53.621576 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:53.621594 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:53.621599 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:53.621667 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.621833 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.621866 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 12:03:53.622267 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.616535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40027 (local address 127.0.203.195:33909)
0619 12:03:53.616822 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.616864 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.617423 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.617711 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.617715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.617724 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:03:53.618515 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.618522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.619417 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.619418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.620247 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.620250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.620614 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.620615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.620760 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.621378 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:03:53.621379 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:03:53.621409 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:03:53.621434 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:03:53.621710 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:03:53.621716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:03:53.621720 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.621823 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.621961 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":150,"threads_started":1}
I20260619 12:03:53.622946 20988 heartbeater.cc:347] Connected to a master server at 127.0.203.254:40027
I20260619 12:03:53.623044 20988 heartbeater.cc:464] Registering TS with master...
I20260619 12:03:53.623261 20988 heartbeater.cc:511] Master 127.0.203.254:40027 requested a full tablet report, sending...
I20260619 12:03:53.623749 20538 ts_manager.cc:194] Registered new tserver with Master: 16ee594cf0b94efb89b346e61da12478 (127.0.203.195:38945)
I20260619 12:03:53.624166 20538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.203.195:33909
I20260619 12:03:53.630146 815 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:03:53.635059 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.631781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.0.203.254:40027)
0619 12:03:53.631841 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.631843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.632010 (+ 167us) server_negotiation.cc:407] Connection header received
0619 12:03:53.632064 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.632065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.632079 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.632140 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:53.632308 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.632920 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.633780 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.633993 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.634059 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.634801 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:53.634802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:53.634802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:53.634806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:53.634823 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:53.634827 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:53.634909 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.634972 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.634996 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 12:03:53.636927 20538 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:03:53.638715 20538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:03:53.646303 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.641546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:45547 (local address 127.0.0.1:42820)
0619 12:03:53.642236 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.642246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.642321 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.642529 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.642530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.642532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:53.642587 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.642589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.643711 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.643712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.645223 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.645225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.645986 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.645986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.646063 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.646076 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.646158 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.646220 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":71,"thread_start_us":81,"threads_started":1}
I20260619 12:03:53.646317 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.641679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.0.203.193:45547)
0619 12:03:53.641849 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.641854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.642327 (+ 473us) server_negotiation.cc:407] Connection header received
0619 12:03:53.642410 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.642412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.642428 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.642530 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:53.642717 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.643631 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.645310 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.645893 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.645984 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.646094 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.646172 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.646223 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260619 12:03:53.646553 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.641319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:42277 (local address 127.0.0.1:59796)
0619 12:03:53.641881 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.641902 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.641961 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.642164 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.642167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.642172 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:53.642296 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.642299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.643791 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.643792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.645084 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.645086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.646217 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.646218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.646297 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.646305 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.646358 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.646436 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":94,"thread_start_us":85,"threads_started":1}
I20260619 12:03:53.646831 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.641715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.0.203.194:42277)
0619 12:03:53.641904 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.641908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.641959 (+ 51us) server_negotiation.cc:407] Connection header received
0619 12:03:53.642029 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.642031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.642046 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.642137 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:53.642413 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.643679 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.645164 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.646002 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.646078 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.646173 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.646686 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.646731 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260619 12:03:53.647033 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.641271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:38945 (local address 127.0.0.1:48408)
0619 12:03:53.641452 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.641497 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.641582 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.642164 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.642167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.642169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:53.642296 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.642300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.643785 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.643786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.645157 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.645160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.646219 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.646219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.646297 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.646307 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.646814 (+ 507us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.646953 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":19,"thread_start_us":108,"threads_started":1}
I20260619 12:03:53.647275 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.641488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.0.203.195:38945)
0619 12:03:53.641942 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.641946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.641957 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:53.642006 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.642009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.642027 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.642137 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:53.642416 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.643716 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.645250 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.646000 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.646077 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.646173 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.647128 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.647198 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":116,"threads_started":1}
I20260619 12:03:53.647220 20660 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0148fc4c544f719613e104d22452f2 (DEFAULT_TABLE table=test-table [id=1df5685c32a141bca167fcea4c281c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:53.647663 20660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0148fc4c544f719613e104d22452f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:53.649498 20922 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0148fc4c544f719613e104d22452f2 (DEFAULT_TABLE table=test-table [id=1df5685c32a141bca167fcea4c281c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:53.649538 20791 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0148fc4c544f719613e104d22452f2 (DEFAULT_TABLE table=test-table [id=1df5685c32a141bca167fcea4c281c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:03:53.649885 20791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0148fc4c544f719613e104d22452f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:53.649897 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0148fc4c544f719613e104d22452f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:53.651499 21002 tablet_bootstrap.cc:492] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac: Bootstrap starting.
I20260619 12:03:53.652348 21002 tablet_bootstrap.cc:654] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:53.652750 21002 log.cc:824] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:53.653331 21004 tablet_bootstrap.cc:492] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Bootstrap starting.
I20260619 12:03:53.653643 21005 tablet_bootstrap.cc:492] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Bootstrap starting.
I20260619 12:03:53.653832 21002 tablet_bootstrap.cc:492] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac: No bootstrap required, opened a new log
I20260619 12:03:53.653947 21002 ts_tablet_manager.cc:1397] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:03:53.654109 21004 tablet_bootstrap.cc:654] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:53.654271 21005 tablet_bootstrap.cc:654] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:53.654474 21004 log.cc:824] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:53.654574 21005 log.cc:824] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:53.655265 21005 tablet_bootstrap.cc:492] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: No bootstrap required, opened a new log
I20260619 12:03:53.655321 21005 ts_tablet_manager.cc:1397] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:03:53.655321 21004 tablet_bootstrap.cc:492] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: No bootstrap required, opened a new log
I20260619 12:03:53.655404 21004 ts_tablet_manager.cc:1397] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:03:53.656886 21002 raft_consensus.cc:348] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.657060 21002 raft_consensus.cc:374] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:53.657095 21002 raft_consensus.cc:729] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116f7f5a0c994ae882922f8667d081ac, State: Initialized, Role: FOLLOWER
I20260619 12:03:53.657234 21002 consensus_queue.cc:260] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.657227 21005 raft_consensus.cc:348] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.657380 21005 raft_consensus.cc:374] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:53.657404 21005 raft_consensus.cc:729] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ee594cf0b94efb89b346e61da12478, State: Initialized, Role: FOLLOWER
I20260619 12:03:53.657533 21002 ts_tablet_manager.cc:1428] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260619 12:03:53.657562 21005 consensus_queue.cc:260] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.657612 20726 heartbeater.cc:503] Master 127.0.203.254:40027 was elected leader, sending a full tablet report...
I20260619 12:03:53.657842 21005 ts_tablet_manager.cc:1428] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 12:03:53.657976 20988 heartbeater.cc:503] Master 127.0.203.254:40027 was elected leader, sending a full tablet report...
I20260619 12:03:53.658079 21004 raft_consensus.cc:348] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.658212 21004 raft_consensus.cc:374] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:53.658245 21004 raft_consensus.cc:729] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b87f5419474e2dac78e0e2687a087e, State: Initialized, Role: FOLLOWER
I20260619 12:03:53.658612 21004 consensus_queue.cc:260] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.658964 21004 ts_tablet_manager.cc:1428] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 12:03:53.659116 20857 heartbeater.cc:503] Master 127.0.203.254:40027 was elected leader, sending a full tablet report...
W20260619 12:03:53.730322 20858 tablet.cc:2369] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:03:53.839088 20727 tablet.cc:2369] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:03:53.866269 20989 tablet.cc:2369] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:03:53.869086 21008 raft_consensus.cc:482] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:03:53.869257 21008 raft_consensus.cc:504] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.869669 21008 leader_election.cc:288] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16ee594cf0b94efb89b346e61da12478 (127.0.203.195:38945), 93b87f5419474e2dac78e0e2687a087e (127.0.203.194:42277)
I20260619 12:03:53.873165 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.870120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:35783 (local address 127.0.203.195:38945)
0619 12:03:53.870212 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.870216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.870306 (+ 90us) server_negotiation.cc:407] Connection header received
0619 12:03:53.870397 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.870399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.870412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.870478 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:53.870688 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.871223 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.872382 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.872794 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.872876 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.872885 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.873042 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.873083 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:03:53.873246 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.870027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.203.193:47763 (local address 127.0.203.194:42277)
0619 12:03:53.870113 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.870116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.870270 (+ 154us) server_negotiation.cc:407] Connection header received
0619 12:03:53.870356 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.870357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.870374 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.870454 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:03:53.870640 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.871203 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.872385 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.872836 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.872916 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.872927 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.873103 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.873151 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:03:53.873311 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.869953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.194:42277 (local address 127.0.203.193:47763)
0619 12:03:53.870179 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.870194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.870281 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.870464 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.870465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.870468 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:53.870566 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.870569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.871300 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.871301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.872290 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.872292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.872932 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.872933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.873018 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.873022 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.873097 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.873147 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":32,"thread_start_us":96,"threads_started":1}
I20260619 12:03:53.873186 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.869955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.195:38945 (local address 127.0.203.193:35783)
0619 12:03:53.870203 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.870216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:03:53.870327 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:03:53.870485 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:03:53.870486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:03:53.870489 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:03:53.870615 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.870618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.871310 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.871311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.872279 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:03:53.872281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.872879 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.872879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:03:53.872964 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.872969 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:03:53.873036 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:03:53.873092 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":44,"thread_start_us":86,"threads_started":1}
I20260619 12:03:53.873749 20811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0148fc4c544f719613e104d22452f2" candidate_uuid: "116f7f5a0c994ae882922f8667d081ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b87f5419474e2dac78e0e2687a087e" is_pre_election: true
I20260619 12:03:53.873843 20942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0148fc4c544f719613e104d22452f2" candidate_uuid: "116f7f5a0c994ae882922f8667d081ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ee594cf0b94efb89b346e61da12478" is_pre_election: true
I20260619 12:03:53.873955 20811 raft_consensus.cc:2454] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 116f7f5a0c994ae882922f8667d081ac in term 0.
I20260619 12:03:53.874042 20942 raft_consensus.cc:2454] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 116f7f5a0c994ae882922f8667d081ac in term 0.
I20260619 12:03:53.874269 20615 leader_election.cc:302] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116f7f5a0c994ae882922f8667d081ac, 93b87f5419474e2dac78e0e2687a087e; no voters:
I20260619 12:03:53.874436 21008 raft_consensus.cc:2781] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:03:53.874503 21008 raft_consensus.cc:482] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:03:53.874522 21008 raft_consensus.cc:3037] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:53.875460 21008 raft_consensus.cc:504] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.875638 21008 leader_election.cc:288] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [CANDIDATE]: Term 1 election: Requested vote from peers 16ee594cf0b94efb89b346e61da12478 (127.0.203.195:38945), 93b87f5419474e2dac78e0e2687a087e (127.0.203.194:42277)
I20260619 12:03:53.875823 20942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0148fc4c544f719613e104d22452f2" candidate_uuid: "116f7f5a0c994ae882922f8667d081ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ee594cf0b94efb89b346e61da12478"
I20260619 12:03:53.875883 20811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0148fc4c544f719613e104d22452f2" candidate_uuid: "116f7f5a0c994ae882922f8667d081ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b87f5419474e2dac78e0e2687a087e"
I20260619 12:03:53.875976 20811 raft_consensus.cc:3037] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:53.875954 20942 raft_consensus.cc:3037] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:53.877024 20811 raft_consensus.cc:2454] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 116f7f5a0c994ae882922f8667d081ac in term 1.
I20260619 12:03:53.877036 20942 raft_consensus.cc:2454] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 116f7f5a0c994ae882922f8667d081ac in term 1.
I20260619 12:03:53.877243 20613 leader_election.cc:302] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116f7f5a0c994ae882922f8667d081ac, 16ee594cf0b94efb89b346e61da12478; no voters:
I20260619 12:03:53.877349 21008 raft_consensus.cc:2781] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:53.877512 21008 raft_consensus.cc:686] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [term 1 LEADER]: Becoming Leader. State: Replica: 116f7f5a0c994ae882922f8667d081ac, State: Running, Role: LEADER
I20260619 12:03:53.877627 21008 consensus_queue.cc:237] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } }
I20260619 12:03:53.878587 20537 catalog_manager.cc:5495] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 116f7f5a0c994ae882922f8667d081ac (127.0.203.193). New cstate: current_term: 1 leader_uuid: "116f7f5a0c994ae882922f8667d081ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "116f7f5a0c994ae882922f8667d081ac" member_type: VOTER last_known_addr { host: "127.0.203.193" port: 45547 } health_report { overall_health: HEALTHY } } }
I20260619 12:03:53.938578 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.935872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.0.203.193:45547)
0619 12:03:53.935937 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:53.935939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:53.935950 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:03:53.935995 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:53.935996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:53.936020 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:53.936145 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:03:53.936361 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.936915 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.937726 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:53.937926 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:53.937983 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:53.938213 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:03:53.938329 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:03:53.938405 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:53.938493 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 12:03:53.938513 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 12:03:53.940915 20942 raft_consensus.cc:1261] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 1 FOLLOWER]: Refusing update from remote peer 116f7f5a0c994ae882922f8667d081ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:53.940956 20811 raft_consensus.cc:1261] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 1 FOLLOWER]: Refusing update from remote peer 116f7f5a0c994ae882922f8667d081ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:03:53.941385 21008 consensus_queue.cc:1035] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ee594cf0b94efb89b346e61da12478" member_type: VOTER last_known_addr { host: "127.0.203.195" port: 38945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:53.941633 21013 consensus_queue.cc:1035] T 1b0148fc4c544f719613e104d22452f2 P 116f7f5a0c994ae882922f8667d081ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b87f5419474e2dac78e0e2687a087e" member_type: VOTER last_known_addr { host: "127.0.203.194" port: 42277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:03:53.944217 21017 mvcc.cc:205] Tried to move back new op lower bound from 7298542116618457088 to 7298542116363534336. Current Snapshot: MvccSnapshot[applied={T|T < 7298542116618457088}]
I20260619 12:03:53.944231 21016 mvcc.cc:205] Tried to move back new op lower bound from 7298542116618457088 to 7298542116363534336. Current Snapshot: MvccSnapshot[applied={T|T < 7298542116618457088}]
I20260619 12:03:53.945164 21019 mvcc.cc:205] Tried to move back new op lower bound from 7298542116618457088 to 7298542116363534336. Current Snapshot: MvccSnapshot[applied={T|T < 7298542116618457088}]
I20260619 12:03:53.947093 20538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:53.947206 20538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:03:53.948081 20538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 038b6e52e20841239eb5097cec8b06df: Sending DeleteTablet for 3 replicas of tablet 1b0148fc4c544f719613e104d22452f2
I20260619 12:03:53.948486 20791 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0148fc4c544f719613e104d22452f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:53 UTC) from {username='slave'} at 127.0.0.1:59796
I20260619 12:03:53.948586 20922 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0148fc4c544f719613e104d22452f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:53 UTC) from {username='slave'} at 127.0.0.1:48408
I20260619 12:03:53.948771 20660 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0148fc4c544f719613e104d22452f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:03:53 UTC) from {username='slave'} at 127.0.0.1:42820
I20260619 12:03:53.948894 21026 tablet_replica.cc:330] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: stopping tablet replica
I20260619 12:03:53.949002 21025 tablet_replica.cc:330] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: stopping tablet replica
I20260619 12:03:53.949012 21026 raft_consensus.cc:2229] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:53.949126 21026 raft_consensus.cc:2258] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:53.949120 21025 raft_consensus.cc:2229] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:03:53.949245 21025 raft_consensus.cc:2258] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:03:53.949319 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 20598
I20260619 12:03:53.950314 21026 ts_tablet_manager.cc:1905] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:53.952170 21026 ts_tablet_manager.cc:1918] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:53.952236 21026 log.cc:1196] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-2/wal/wals/1b0148fc4c544f719613e104d22452f2
I20260619 12:03:53.952514 21026 ts_tablet_manager.cc:1939] T 1b0148fc4c544f719613e104d22452f2 P 16ee594cf0b94efb89b346e61da12478: Deleting consensus metadata
I20260619 12:03:53.953022 20522 catalog_manager.cc:4841] TS 16ee594cf0b94efb89b346e61da12478 (127.0.203.195:38945): tablet 1b0148fc4c544f719613e104d22452f2 (table test-table [id=1df5685c32a141bca167fcea4c281c64]) successfully deleted
I20260619 12:03:53.953099 21025 ts_tablet_manager.cc:1905] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:03:53.954746 21025 ts_tablet_manager.cc:1918] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:03:53.954893 21025 log.cc:1196] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Deleting WAL directory at /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781870380032316-815-0/minicluster-data/ts-1/wal/wals/1b0148fc4c544f719613e104d22452f2
I20260619 12:03:53.955155 21025 ts_tablet_manager.cc:1939] T 1b0148fc4c544f719613e104d22452f2 P 93b87f5419474e2dac78e0e2687a087e: Deleting consensus metadata
I20260619 12:03:53.955648 20524 catalog_manager.cc:4841] TS 93b87f5419474e2dac78e0e2687a087e (127.0.203.194:42277): tablet 1b0148fc4c544f719613e104d22452f2 (table test-table [id=1df5685c32a141bca167fcea4c281c64]) successfully deleted
W20260619 12:03:53.956195 20522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.203.193:45547 (error 108)
I20260619 12:03:53.956506 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 20729
I20260619 12:03:53.957437 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:53.956743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.193:45547 (local address 127.0.0.1:42838)
0619 12:03:53.957344 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 12:03:53.957390 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.193:45547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567}
W20260619 12:03:53.957621 20522 catalog_manager.cc:4584] TS 116f7f5a0c994ae882922f8667d081ac (127.0.203.193:45547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b0148fc4c544f719613e104d22452f2: Network error: Client connection negotiation failed: client connection to 127.0.203.193:45547: connect: Connection refused (error 111)
I20260619 12:03:53.964185 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 20860
I20260619 12:03:53.970228 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 20507
2026-06-19T12:03:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (962 ms)
[----------] 6 tests from EncryptionPolicyTest (9391 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:54 dist-test-slave-fbq8 krb5kdc[21031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:54 dist-test-slave-fbq8 krb5kdc[21031](info): set up 2 sockets
Jun 19 12:03:54 dist-test-slave-fbq8 krb5kdc[21031](info): commencing operation
krb5kdc: starting...
W20260619 12:03:55.045070 815 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:03:55 dist-test-slave-fbq8 krb5kdc[21031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:03:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:03:55.181634 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:55.273976 21044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:55.274230 21044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:55.274253 21044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:55.275373 21044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:55.275414 21044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:55.275429 21044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:55.275444 21044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:55.275457 21044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:55.276944 21044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41461
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:55.277168 21044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:55.277359 21044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:55.282859 21049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:55.282879 21052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:55.282879 21050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:55.282980 21044 server_base.cc:1034] running on GCE node
I20260619 12:03:55.283478 21044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:55.283857 21044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:55.285110 21044 hybrid_clock.cc:648] HybridClock initialized: now 1781870635285077 us; error 53 us; skew 500 ppm
Jun 19 12:03:55 dist-test-slave-fbq8 krb5kdc[21031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:55.292455 21044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:03:55.293134 21044 webserver.cc:460] Webserver started at http://127.0.0.1:45077/ using document root <none> and password file <none>
I20260619 12:03:55.293390 21044 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:55.293466 21044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:55.293620 21044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:55.294814 21044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "15e57769e37f4342865c8872c91059f0"
format_stamp: "Formatted at 2026-06-19 12:03:55 on dist-test-slave-fbq8"
server_key: "960616849367d20e0a30e453fa878faa"
server_key_iv: "9264c35cdd9f56ed07285a00c09ca1e3"
server_key_version: "encryptionkey@0"
I20260619 12:03:55.295343 21044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "15e57769e37f4342865c8872c91059f0"
format_stamp: "Formatted at 2026-06-19 12:03:55 on dist-test-slave-fbq8"
server_key: "960616849367d20e0a30e453fa878faa"
server_key_iv: "9264c35cdd9f56ed07285a00c09ca1e3"
server_key_version: "encryptionkey@0"
I20260619 12:03:55.297135 21044 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 12:03:55.298146 21059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:55.298424 21044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:55.298542 21044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "15e57769e37f4342865c8872c91059f0"
format_stamp: "Formatted at 2026-06-19 12:03:55 on dist-test-slave-fbq8"
server_key: "960616849367d20e0a30e453fa878faa"
server_key_iv: "9264c35cdd9f56ed07285a00c09ca1e3"
server_key_version: "encryptionkey@0"
I20260619 12:03:55.298655 21044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:55.332041 21044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:55.334645 21044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:55.334830 21044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:55.339679 21044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41461
I20260619 12:03:55.339850 21111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41461 every 8 connection(s)
I20260619 12:03:55.340164 21044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:55.341356 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:55.344045 21112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0: Bootstrap starting.
I20260619 12:03:55.344766 21112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:55.345106 21112 log.cc:824] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:55.346066 21112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0: No bootstrap required, opened a new log
I20260619 12:03:55.347885 21112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e57769e37f4342865c8872c91059f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260619 12:03:55.348040 21112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:55.348065 21112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e57769e37f4342865c8872c91059f0, State: Initialized, Role: FOLLOWER
I20260619 12:03:55.348251 21112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e57769e37f4342865c8872c91059f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260619 12:03:55.348366 21112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:55.348405 21112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:55.348470 21112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:55.348486 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21044
I20260619 12:03:55.348613 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:55.348872 815 external_mini_cluster.cc:1428] Setting key bc2c3caeb94df824201ace79d0ada580
I20260619 12:03:55.349262 21112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e57769e37f4342865c8872c91059f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260619 12:03:55.349403 21112 leader_election.cc:302] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15e57769e37f4342865c8872c91059f0; no voters:
I20260619 12:03:55.349604 21112 leader_election.cc:288] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:55.349700 21117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:55.349889 21112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:55.349947 21117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [term 1 LEADER]: Becoming Leader. State: Replica: 15e57769e37f4342865c8872c91059f0, State: Running, Role: LEADER
I20260619 12:03:55.350080 21117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e57769e37f4342865c8872c91059f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } }
I20260619 12:03:55.350528 21117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15e57769e37f4342865c8872c91059f0. Latest consensus state: current_term: 1 leader_uuid: "15e57769e37f4342865c8872c91059f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e57769e37f4342865c8872c91059f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } } }
I20260619 12:03:55.350636 21117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:55.350539 21119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15e57769e37f4342865c8872c91059f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e57769e37f4342865c8872c91059f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41461 } } }
I20260619 12:03:55.350780 21119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:55.350906 21123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:55.351467 21123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:55.353503 21123 catalog_manager.cc:1347] Generated new cluster ID: 153463843b974712a07ba9df3e7b507e
I20260619 12:03:55.353569 21123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:03:55 dist-test-slave-fbq8 krb5kdc[21031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:03:55.362675 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:55.349448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50679 (local address 127.0.0.1:41461)
0619 12:03:55.349729 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:55.349741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:55.349781 (+ 40us) server_negotiation.cc:407] Connection header received
0619 12:03:55.349911 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:55.349920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:55.349983 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:55.351189 (+ 1206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:55.351330 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:55.352950 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:55.354105 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:55.354734 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:55.354845 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:55.357987 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:55.357994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:55.357994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:55.358011 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:55.360282 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:55.360831 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:55.360835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:55.360835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:55.360977 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:55.361234 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:55.361235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:55.361235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:55.361612 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:55.361801 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:55.362284 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 12:03:55.362411 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
I20260619 12:03:55.364228 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:03:55.370319 21123 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:55.370893 21123 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:55.377031 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:55.365970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35673 (local address 127.0.0.1:41461)
0619 12:03:55.366061 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:55.366064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:55.366273 (+ 209us) server_negotiation.cc:407] Connection header received
0619 12:03:55.366320 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:55.366321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:55.366335 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:55.366426 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:55.367101 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:55.367935 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:55.369289 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:55.369653 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:55.369736 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:55.373734 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:55.373736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:55.373737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:55.373747 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:55.375590 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:55.376083 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:55.376085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:55.376085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:55.376159 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:55.376399 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:55.376400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:55.376401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:55.376553 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:55.376735 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:55.376853 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 12:03:55.376926 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 12:03:55.381479 21123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15e57769e37f4342865c8872c91059f0: Generated new TSK 0
I20260619 12:03:55.381738 21123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:03:55.385437 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21044
W20260619 12:03:55.390223 21029 connection.cc:535] client connection to 127.0.0.1:41461 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41461): Connection reset by peer (error 104)
2026-06-19T12:03:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:55 dist-test-slave-fbq8 krb5kdc[21143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:55 dist-test-slave-fbq8 krb5kdc[21143](info): set up 2 sockets
Jun 19 12:03:55 dist-test-slave-fbq8 krb5kdc[21143](info): commencing operation
krb5kdc: starting...
W20260619 12:03:56.441601 815 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:03:56 dist-test-slave-fbq8 krb5kdc[21143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:03:56.576227 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:56.667837 21156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:56.668164 21156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:56.668210 21156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:56.669799 21156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:56.669904 21156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:56.669934 21156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:56.669955 21156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:56.669991 21156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:56.671720 21156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:56.672055 21156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:56.672330 21156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:56.678094 21162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:56.678099 21161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:56.678118 21156 server_base.cc:1034] running on GCE node
W20260619 12:03:56.678100 21164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:56.678735 21156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:56.679189 21156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:56.680377 21156 hybrid_clock.cc:648] HybridClock initialized: now 1781870636680339 us; error 67 us; skew 500 ppm
Jun 19 12:03:56 dist-test-slave-fbq8 krb5kdc[21143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:56.683457 21156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:03:56.684230 21156 webserver.cc:460] Webserver started at http://127.0.0.1:46037/ using document root <none> and password file <none>
I20260619 12:03:56.684459 21156 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:56.684507 21156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:56.684612 21156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:56.685786 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "31150c1d68674e6abdadf336713444c7"
format_stamp: "Formatted at 2026-06-19 12:03:56 on dist-test-slave-fbq8"
server_key: "7550d54dae87ff4f73f87e716854d4ea"
server_key_iv: "ee686c4d5baa9899e89f15a6db93aefc"
server_key_version: "encryptionkey@0"
I20260619 12:03:56.686334 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "31150c1d68674e6abdadf336713444c7"
format_stamp: "Formatted at 2026-06-19 12:03:56 on dist-test-slave-fbq8"
server_key: "7550d54dae87ff4f73f87e716854d4ea"
server_key_iv: "ee686c4d5baa9899e89f15a6db93aefc"
server_key_version: "encryptionkey@0"
I20260619 12:03:56.688172 21156 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:03:56.689196 21171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:56.689543 21156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:03:56.689702 21156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "31150c1d68674e6abdadf336713444c7"
format_stamp: "Formatted at 2026-06-19 12:03:56 on dist-test-slave-fbq8"
server_key: "7550d54dae87ff4f73f87e716854d4ea"
server_key_iv: "ee686c4d5baa9899e89f15a6db93aefc"
server_key_version: "encryptionkey@0"
I20260619 12:03:56.689810 21156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:56.699954 21156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:56.702630 21156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:56.702802 21156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:56.707688 21156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42043
I20260619 12:03:56.707732 21223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42043 every 8 connection(s)
I20260619 12:03:56.708214 21156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:56.709259 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:56.711637 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7: Bootstrap starting.
I20260619 12:03:56.712245 21224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:56.712486 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21156
I20260619 12:03:56.712544 21224 log.cc:824] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:56.712643 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:56.712939 815 external_mini_cluster.cc:1428] Setting key 5f7aff6784add56559d2545b427efec0
I20260619 12:03:56.713564 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7: No bootstrap required, opened a new log
I20260619 12:03:56.716389 21224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31150c1d68674e6abdadf336713444c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260619 12:03:56.716562 21224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:56.716591 21224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31150c1d68674e6abdadf336713444c7, State: Initialized, Role: FOLLOWER
I20260619 12:03:56.716784 21224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31150c1d68674e6abdadf336713444c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260619 12:03:56.716888 21224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:56.716933 21224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:56.716990 21224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:56.717556 21224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31150c1d68674e6abdadf336713444c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260619 12:03:56.717644 21224 leader_election.cc:302] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31150c1d68674e6abdadf336713444c7; no voters:
I20260619 12:03:56.717881 21224 leader_election.cc:288] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:56.717911 21229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:56.718042 21229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [term 1 LEADER]: Becoming Leader. State: Replica: 31150c1d68674e6abdadf336713444c7, State: Running, Role: LEADER
I20260619 12:03:56.718160 21229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31150c1d68674e6abdadf336713444c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } }
I20260619 12:03:56.718186 21224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:56.718533 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31150c1d68674e6abdadf336713444c7. Latest consensus state: current_term: 1 leader_uuid: "31150c1d68674e6abdadf336713444c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31150c1d68674e6abdadf336713444c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } } }
I20260619 12:03:56.718840 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:56.718763 21231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31150c1d68674e6abdadf336713444c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31150c1d68674e6abdadf336713444c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42043 } } }
I20260619 12:03:56.719147 21231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:56.719843 21238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:56.720479 21238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:56.720682 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:56.713585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47313 (local address 127.0.0.1:42043)
0619 12:03:56.713808 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:56.713819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:56.713857 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:03:56.714823 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:56.714833 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:56.714896 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:56.715065 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:56.715285 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:56.716759 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:56.717709 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:56.718515 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:56.718648 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:56.719621 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:56.719625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:56.719625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:56.719854 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:56.719901 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:56.719918 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:56.720002 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:56.720198 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 12:03:56.720231 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
I20260619 12:03:56.722594 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:03:56.724407 21238 catalog_manager.cc:1347] Generated new cluster ID: 04ed1a8bb46f4e99bced249c6a58b7da
I20260619 12:03:56.724486 21238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:56.732240 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:56.724602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41365 (local address 127.0.0.1:42043)
0619 12:03:56.724669 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:56.724671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:56.728903 (+ 4232us) server_negotiation.cc:407] Connection header received
0619 12:03:56.729070 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:56.729071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:56.729082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:56.729155 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:56.729324 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:56.729978 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:56.730789 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:56.731025 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:56.731218 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:56.731947 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:56.731948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:56.731949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:56.731954 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:56.731970 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:56.731974 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:56.732046 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:56.732118 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 12:03:56.732128 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 12:03:56.762508 21238 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:56.763151 21238 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:56.768560 21238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31150c1d68674e6abdadf336713444c7: Generated new TSK 0
I20260619 12:03:56.768818 21238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:03:56.779066 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21156
W20260619 12:03:56.783996 21141 connection.cc:535] client connection to 127.0.0.1:42043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42043): Connection reset by peer (error 104)
2026-06-19T12:03:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:56 dist-test-slave-fbq8 krb5kdc[21255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:56 dist-test-slave-fbq8 krb5kdc[21255](info): set up 2 sockets
Jun 19 12:03:56 dist-test-slave-fbq8 krb5kdc[21255](info): commencing operation
krb5kdc: starting...
W20260619 12:03:57.877328 815 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:03:57 dist-test-slave-fbq8 krb5kdc[21255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:03:58.020037 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:58.116384 21268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:58.117218 21268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:58.117300 21268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:58.119349 21268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:58.119415 21268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:58.119431 21268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:58.119444 21268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:58.119460 21268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:58.121578 21268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:58.121949 21268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:58.122212 21268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:58.128253 21274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:58.128539 21273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:58.128706 21276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:58.128875 21268 server_base.cc:1034] running on GCE node
I20260619 12:03:58.129508 21268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:58.129981 21268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:58.131196 21268 hybrid_clock.cc:648] HybridClock initialized: now 1781870638131169 us; error 49 us; skew 500 ppm
Jun 19 12:03:58 dist-test-slave-fbq8 krb5kdc[21255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:58.134526 21268 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:03:58.135123 21268 webserver.cc:460] Webserver started at http://127.0.0.1:40677/ using document root <none> and password file <none>
I20260619 12:03:58.135406 21268 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:58.135473 21268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:58.135633 21268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:58.136999 21268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "349c75193f5443c29bcaf6fe46bcdcd9"
format_stamp: "Formatted at 2026-06-19 12:03:58 on dist-test-slave-fbq8"
server_key: "304b4f0749e17f5ae1338f2cf639a0ca"
server_key_iv: "0c0435dcf2cef3b77e64f4be42ab2f18"
server_key_version: "encryptionkey@0"
I20260619 12:03:58.137391 21268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "349c75193f5443c29bcaf6fe46bcdcd9"
format_stamp: "Formatted at 2026-06-19 12:03:58 on dist-test-slave-fbq8"
server_key: "304b4f0749e17f5ae1338f2cf639a0ca"
server_key_iv: "0c0435dcf2cef3b77e64f4be42ab2f18"
server_key_version: "encryptionkey@0"
I20260619 12:03:58.139086 21268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:58.140098 21283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:58.140571 21268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:58.140784 21268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "349c75193f5443c29bcaf6fe46bcdcd9"
format_stamp: "Formatted at 2026-06-19 12:03:58 on dist-test-slave-fbq8"
server_key: "304b4f0749e17f5ae1338f2cf639a0ca"
server_key_iv: "0c0435dcf2cef3b77e64f4be42ab2f18"
server_key_version: "encryptionkey@0"
I20260619 12:03:58.140915 21268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:58.144829 21268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:58.147738 21268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:58.147924 21268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:58.153542 21268 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42527
I20260619 12:03:58.153612 21335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42527 every 8 connection(s)
I20260619 12:03:58.154033 21268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:58.155236 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:58.156138 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21268
I20260619 12:03:58.156301 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:58.156528 815 external_mini_cluster.cc:1428] Setting key 1a61652d63cb5570cb19a506dc138ae0
I20260619 12:03:58.158663 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9: Bootstrap starting.
I20260619 12:03:58.159361 21336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:58.159716 21336 log.cc:824] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:58.159881 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:58.157332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44443 (local address 127.0.0.1:42527)
0619 12:03:58.157590 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:58.157601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:58.157639 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:03:58.158343 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:58.158353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:58.158391 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:58.158591 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:58.159414 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:58.159418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:58.159418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:58.159423 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:58.159471 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:58.159487 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:58.159603 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:58.159635 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 12:03:58.159681 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260619 12:03:58.160547 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9: No bootstrap required, opened a new log
I20260619 12:03:58.161662 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:03:58.164935 21336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42527 } }
I20260619 12:03:58.165200 21336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:58.165282 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:58.163559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38831 (local address 127.0.0.1:42527)
0619 12:03:58.163668 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:58.163671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:58.163689 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:03:58.163709 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:58.163710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:58.163714 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:58.163807 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:58.164957 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:58.164959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:58.164960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:03:58.164965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:58.164986 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:03:58.164991 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:58.165055 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:58.165119 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 12:03:58.165136 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 12:03:58.165270 21336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 349c75193f5443c29bcaf6fe46bcdcd9, State: Initialized, Role: FOLLOWER
I20260619 12:03:58.165585 21336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42527 } }
I20260619 12:03:58.165747 21336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:58.165802 21336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:58.165859 21336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:58.166723 21336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42527 } }
I20260619 12:03:58.166837 21336 leader_election.cc:302] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 349c75193f5443c29bcaf6fe46bcdcd9; no voters:
I20260619 12:03:58.167025 21336 leader_election.cc:288] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:58.167140 21346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:58.167367 21336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:58.167387 21346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [term 1 LEADER]: Becoming Leader. State: Replica: 349c75193f5443c29bcaf6fe46bcdcd9, State: Running, Role: LEADER
I20260619 12:03:58.167712 21346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42527 } }
I20260619 12:03:58.168666 21347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42527 } } }
I20260619 12:03:58.168785 21347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:58.169059 21348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 349c75193f5443c29bcaf6fe46bcdcd9. Latest consensus state: current_term: 1 leader_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "349c75193f5443c29bcaf6fe46bcdcd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42527 } } }
I20260619 12:03:58.169054 21355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:03:58.169181 21348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:58.169823 21355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:58.172514 21355 catalog_manager.cc:1347] Generated new cluster ID: 33533193ccb74e2d9c53f38472df2518
I20260619 12:03:58.172600 21355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:58.182107 21355 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:58.182364 21355 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:58.192680 21355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 349c75193f5443c29bcaf6fe46bcdcd9: Generated new TSK 0
I20260619 12:03:58.192961 21355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:03:58.196483 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21268
2026-06-19T12:03:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:58 dist-test-slave-fbq8 krb5kdc[21367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:58 dist-test-slave-fbq8 krb5kdc[21367](info): set up 2 sockets
Jun 19 12:03:58 dist-test-slave-fbq8 krb5kdc[21367](info): commencing operation
krb5kdc: starting...
W20260619 12:03:59.288955 815 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:03:59 dist-test-slave-fbq8 krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:03:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:03:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:03:59.423892 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:03:59.511533 21380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:03:59.511766 21380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:03:59.511793 21380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:03:59.513195 21380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:03:59.513257 21380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:03:59.513273 21380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:03:59.513288 21380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:03:59.513302 21380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:03:59.515134 21380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:03:59.515455 21380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:03:59.515726 21380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:03:59.521358 21386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:03:59.521358 21388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:59.521504 21380 server_base.cc:1034] running on GCE node
W20260619 12:03:59.521358 21385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:03:59.522020 21380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:03:59.522408 21380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:03:59.523602 21380 hybrid_clock.cc:648] HybridClock initialized: now 1781870639523580 us; error 56 us; skew 500 ppm
Jun 19 12:03:59 dist-test-slave-fbq8 krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:03:59.526681 21380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:03:59.527192 21380 webserver.cc:460] Webserver started at http://127.0.0.1:34991/ using document root <none> and password file <none>
I20260619 12:03:59.527480 21380 fs_manager.cc:359] Metadata directory not provided
I20260619 12:03:59.527537 21380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:03:59.527683 21380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:03:59.529086 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "133d39f4df934ffab9a590ee98675465"
format_stamp: "Formatted at 2026-06-19 12:03:59 on dist-test-slave-fbq8"
server_key: "82f7c8047815abfa2fff6ed1da3c5772"
server_key_iv: "3e41171295993556c357e59027a13375"
server_key_version: "encryptionkey@0"
I20260619 12:03:59.529474 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "133d39f4df934ffab9a590ee98675465"
format_stamp: "Formatted at 2026-06-19 12:03:59 on dist-test-slave-fbq8"
server_key: "82f7c8047815abfa2fff6ed1da3c5772"
server_key_iv: "3e41171295993556c357e59027a13375"
server_key_version: "encryptionkey@0"
I20260619 12:03:59.531061 21380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:03:59.532023 21395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:03:59.532332 21380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:03:59.532421 21380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "133d39f4df934ffab9a590ee98675465"
format_stamp: "Formatted at 2026-06-19 12:03:59 on dist-test-slave-fbq8"
server_key: "82f7c8047815abfa2fff6ed1da3c5772"
server_key_iv: "3e41171295993556c357e59027a13375"
server_key_version: "encryptionkey@0"
I20260619 12:03:59.532518 21380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:03:59.558578 21380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:03:59.561048 21380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:03:59.561223 21380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:03:59.566248 21380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46047
I20260619 12:03:59.566448 21447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46047 every 8 connection(s)
I20260619 12:03:59.566720 21380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:03:59.567785 21448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:03:59.570389 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21380
I20260619 12:03:59.570540 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:03:59.570621 21448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465: Bootstrap starting.
I20260619 12:03:59.570755 815 external_mini_cluster.cc:1428] Setting key a8dde22e523f81d005d544fbf0167d58
I20260619 12:03:59.571393 21448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465: Neither blocks nor log segments found. Creating new log.
I20260619 12:03:59.571775 21448 log.cc:824] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465: Log is configured to *not* fsync() on all Append() calls
I20260619 12:03:59.572932 21448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465: No bootstrap required, opened a new log
I20260619 12:03:59.575412 21448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133d39f4df934ffab9a590ee98675465" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46047 } }
I20260619 12:03:59.575636 21448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:03:59.575672 21448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133d39f4df934ffab9a590ee98675465, State: Initialized, Role: FOLLOWER
I20260619 12:03:59.575827 21448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133d39f4df934ffab9a590ee98675465" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46047 } }
I20260619 12:03:59.575927 21448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:03:59.575963 21448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:03:59.576018 21448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:03:59.576838 21448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133d39f4df934ffab9a590ee98675465" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46047 } }
I20260619 12:03:59.576979 21448 leader_election.cc:302] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133d39f4df934ffab9a590ee98675465; no voters:
I20260619 12:03:59.577208 21448 leader_election.cc:288] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:03:59.577410 21454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:03:59.577569 21448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:03:59.577608 21454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [term 1 LEADER]: Becoming Leader. State: Replica: 133d39f4df934ffab9a590ee98675465, State: Running, Role: LEADER
I20260619 12:03:59.577785 21454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133d39f4df934ffab9a590ee98675465" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46047 } }
I20260619 12:03:59.578625 21453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "133d39f4df934ffab9a590ee98675465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133d39f4df934ffab9a590ee98675465" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46047 } } }
I20260619 12:03:59.578771 21453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:59.579025 21455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133d39f4df934ffab9a590ee98675465. Latest consensus state: current_term: 1 leader_uuid: "133d39f4df934ffab9a590ee98675465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133d39f4df934ffab9a590ee98675465" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46047 } } }
I20260619 12:03:59.579119 21455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465 [sys.catalog]: This master's current role is: LEADER
I20260619 12:03:59.579303 21463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 12:03:59 dist-test-slave-fbq8 krb5kdc[21367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:03:59.579908 21463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:03:59.583025 21470 mvcc.cc:205] Tried to move back new op lower bound from 7298542139724713984 to 7298542139711721472. Current Snapshot: MvccSnapshot[applied={T|T < 7298542139724713984}]
I20260619 12:03:59.583643 21463 catalog_manager.cc:1347] Generated new cluster ID: f957548679ca497382d262cb39e706ea
I20260619 12:03:59.583716 21463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:03:59.584161 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:59.571439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50293 (local address 127.0.0.1:46047)
0619 12:03:59.571643 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:59.571654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:59.571687 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:03:59.572522 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:59.572531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:59.572594 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:59.572804 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:59.573742 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:59.574745 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:59.575890 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:59.576455 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:59.576560 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:59.580107 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:59.580113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:59.580114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:59.580134 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:59.582085 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:59.582610 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:59.582614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:59.582615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:59.582734 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:59.583002 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:59.583004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:59.583004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:59.583436 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:59.583625 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:59.583804 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 12:03:59.583963 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260619 12:03:59.585964 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:03:59.593106 21463 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:03:59.593878 21463 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:03:59.595239 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:03:59.587830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48909 (local address 127.0.0.1:46047)
0619 12:03:59.587873 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 12:03:59.587875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:03:59.588862 (+ 987us) server_negotiation.cc:407] Connection header received
0619 12:03:59.588941 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:03:59.588942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:03:59.588954 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:03:59.589010 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:03:59.589563 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:59.590086 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:59.590800 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:03:59.591050 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:03:59.591130 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:03:59.592433 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:03:59.592434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:03:59.592434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:03:59.592443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:03:59.594017 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:59.594377 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:59.594379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:59.594379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:59.594439 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:03:59.594633 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:03:59.594634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:03:59.594634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:03:59.594763 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:03:59.594862 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:03:59.594998 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 12:03:59.595106 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:03:59.606841 21463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 133d39f4df934ffab9a590ee98675465: Generated new TSK 0
I20260619 12:03:59.607107 21463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:03:59.614773 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21380
2026-06-19T12:03:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:03:59 dist-test-slave-fbq8 krb5kdc[21479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:03:59 dist-test-slave-fbq8 krb5kdc[21479](info): set up 2 sockets
Jun 19 12:03:59 dist-test-slave-fbq8 krb5kdc[21479](info): commencing operation
krb5kdc: starting...
W20260619 12:04:00.673309 815 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:00 dist-test-slave-fbq8 krb5kdc[21479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:00.811817 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:00.898363 21492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:00.898643 21492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:00.898684 21492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:00.899928 21492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:00.900038 21492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:00.900067 21492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:00.900098 21492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:00.900130 21492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:00.901839 21492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:00.902149 21492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:00.902405 21492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:00.908052 21500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:00.908093 21497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:00.908435 21492 server_base.cc:1034] running on GCE node
W20260619 12:04:00.908110 21498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:00.908810 21492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:00.909299 21492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:00.910538 21492 hybrid_clock.cc:648] HybridClock initialized: now 1781870640910521 us; error 36 us; skew 500 ppm
Jun 19 12:04:00 dist-test-slave-fbq8 krb5kdc[21479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:00.917490 21492 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:00.918057 21492 webserver.cc:460] Webserver started at http://127.0.0.1:40107/ using document root <none> and password file <none>
I20260619 12:04:00.918322 21492 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:00.918376 21492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:00.918519 21492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:00.919649 21492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "379ca18505d94227b6a49be99b68c406"
format_stamp: "Formatted at 2026-06-19 12:04:00 on dist-test-slave-fbq8"
server_key: "d70b76b11713122c641827d74969f806"
server_key_iv: "d9d26408d2b93d0c7b1956a755056134"
server_key_version: "encryptionkey@0"
I20260619 12:04:00.920017 21492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "379ca18505d94227b6a49be99b68c406"
format_stamp: "Formatted at 2026-06-19 12:04:00 on dist-test-slave-fbq8"
server_key: "d70b76b11713122c641827d74969f806"
server_key_iv: "d9d26408d2b93d0c7b1956a755056134"
server_key_version: "encryptionkey@0"
I20260619 12:04:00.921691 21492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:04:00.922747 21507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:00.923038 21492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:00.923178 21492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "379ca18505d94227b6a49be99b68c406"
format_stamp: "Formatted at 2026-06-19 12:04:00 on dist-test-slave-fbq8"
server_key: "d70b76b11713122c641827d74969f806"
server_key_iv: "d9d26408d2b93d0c7b1956a755056134"
server_key_version: "encryptionkey@0"
I20260619 12:04:00.923365 21492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:00.933092 21492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:00.936040 21492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:00.936220 21492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:00.941009 21492 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39039
I20260619 12:04:00.941051 21559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39039 every 8 connection(s)
I20260619 12:04:00.941440 21492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:00.942440 21560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:00.945432 21560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406: Bootstrap starting.
I20260619 12:04:00.946182 21560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:00.946518 21560 log.cc:824] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:00.947535 21560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406: No bootstrap required, opened a new log
I20260619 12:04:00.947925 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21492
I20260619 12:04:00.948108 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:00.948330 815 external_mini_cluster.cc:1428] Setting key fd215c9b3d3938064e320dfd6343d22c
I20260619 12:04:00.950093 21560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379ca18505d94227b6a49be99b68c406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260619 12:04:00.950295 21560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:00.950335 21560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379ca18505d94227b6a49be99b68c406, State: Initialized, Role: FOLLOWER
I20260619 12:04:00.950525 21560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379ca18505d94227b6a49be99b68c406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260619 12:04:00.950625 21560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:00.950659 21560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:00.950714 21560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:00.951525 21560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379ca18505d94227b6a49be99b68c406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260619 12:04:00.951653 21560 leader_election.cc:302] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 379ca18505d94227b6a49be99b68c406; no voters:
I20260619 12:04:00.951828 21560 leader_election.cc:288] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:00.952036 21565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:00.952112 21560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:00.952297 21565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [term 1 LEADER]: Becoming Leader. State: Replica: 379ca18505d94227b6a49be99b68c406, State: Running, Role: LEADER
I20260619 12:04:00.952486 21565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379ca18505d94227b6a49be99b68c406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } }
I20260619 12:04:00.953094 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 379ca18505d94227b6a49be99b68c406. Latest consensus state: current_term: 1 leader_uuid: "379ca18505d94227b6a49be99b68c406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379ca18505d94227b6a49be99b68c406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } } }
I20260619 12:04:00.953246 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:00.953574 21577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:00.954053 21567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "379ca18505d94227b6a49be99b68c406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379ca18505d94227b6a49be99b68c406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39039 } } }
I20260619 12:04:00.954221 21567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:00.954751 21577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:00.956266 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:00.949081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56317 (local address 127.0.0.1:39039)
0619 12:04:00.949349 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:00.949361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:00.949401 (+ 40us) server_negotiation.cc:407] Connection header received
0619 12:04:00.950044 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:00.950054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:00.950136 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:00.950321 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:00.950624 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:00.952445 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:00.953764 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:00.954411 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:00.954530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:00.955406 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:00.955410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:00.955411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:00.955674 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:00.955718 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:00.955735 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:00.955809 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:00.955951 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 12:04:00.956023 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":126,"threads_started":1}
I20260619 12:04:00.956835 21577 catalog_manager.cc:1347] Generated new cluster ID: 4b3850727ba94fd48f29b73479eab91a
I20260619 12:04:00.956904 21577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:00.957538 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:00.967954 21577 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:00.968757 21577 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:00.971658 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:00.959452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35645 (local address 127.0.0.1:39039)
0619 12:04:00.959524 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:00.959527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:00.967531 (+ 8004us) server_negotiation.cc:407] Connection header received
0619 12:04:00.967657 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:00.967659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:00.967686 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:00.967746 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:00.967982 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:00.968802 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:00.970027 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:00.970359 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:00.970456 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:00.971321 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:00.971322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:00.971322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:00.971328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:00.971349 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:00.971354 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:00.971430 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:00.971508 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 12:04:00.971565 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:04:00.972750 21577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 379ca18505d94227b6a49be99b68c406: Generated new TSK 0
I20260619 12:04:00.972977 21577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:00.981751 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21492
2026-06-19T12:04:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:01 dist-test-slave-fbq8 krb5kdc[21591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:01 dist-test-slave-fbq8 krb5kdc[21591](info): set up 2 sockets
Jun 19 12:04:01 dist-test-slave-fbq8 krb5kdc[21591](info): commencing operation
krb5kdc: starting...
W20260619 12:04:02.048897 815 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:02 dist-test-slave-fbq8 krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:02.179739 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:02.263278 21604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:02.263510 21604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:02.263536 21604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:02.264894 21604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:02.264950 21604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:02.264966 21604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:02.264979 21604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:02.264994 21604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:02.266657 21604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:02.266952 21604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:02.267210 21604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:02.272750 21609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:02.272861 21604 server_base.cc:1034] running on GCE node
W20260619 12:04:02.272749 21612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:02.272783 21610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:02.273389 21604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:02.273679 21604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:02.274905 21604 hybrid_clock.cc:648] HybridClock initialized: now 1781870642274882 us; error 51 us; skew 500 ppm
Jun 19 12:04:02 dist-test-slave-fbq8 krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:02.277745 21604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:02.278285 21604 webserver.cc:460] Webserver started at http://127.0.0.1:33649/ using document root <none> and password file <none>
I20260619 12:04:02.278570 21604 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:02.278647 21604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:02.278789 21604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:02.280090 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "f57a7ec39d73414a8f09a0574ad4b528"
format_stamp: "Formatted at 2026-06-19 12:04:02 on dist-test-slave-fbq8"
server_key: "4a24a3cad5264f7b6353d614dd115e7d"
server_key_iv: "ed303dcbdcb17d1be7a366a23ce660ed"
server_key_version: "encryptionkey@0"
I20260619 12:04:02.280457 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "f57a7ec39d73414a8f09a0574ad4b528"
format_stamp: "Formatted at 2026-06-19 12:04:02 on dist-test-slave-fbq8"
server_key: "4a24a3cad5264f7b6353d614dd115e7d"
server_key_iv: "ed303dcbdcb17d1be7a366a23ce660ed"
server_key_version: "encryptionkey@0"
I20260619 12:04:02.281993 21604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:04:02.282876 21619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:02.283138 21604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:02.283262 21604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "f57a7ec39d73414a8f09a0574ad4b528"
format_stamp: "Formatted at 2026-06-19 12:04:02 on dist-test-slave-fbq8"
server_key: "4a24a3cad5264f7b6353d614dd115e7d"
server_key_iv: "ed303dcbdcb17d1be7a366a23ce660ed"
server_key_version: "encryptionkey@0"
I20260619 12:04:02.283337 21604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:02.286978 21604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:02.289448 21604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:02.289615 21604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:02.294450 21604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42799
I20260619 12:04:02.294507 21671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42799 every 8 connection(s)
I20260619 12:04:02.294890 21604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:02.295315 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21604
I20260619 12:04:02.295423 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:02.295585 815 external_mini_cluster.cc:1428] Setting key 600e89e0ff0c65514979fc3ef73b7457
I20260619 12:04:02.296890 21672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:02.298686 21675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:02.296245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55947 (local address 127.0.0.1:42799)
0619 12:04:02.296431 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:02.296441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:02.296481 (+ 40us) server_negotiation.cc:407] Connection header received
0619 12:04:02.297111 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:02.297119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:02.297148 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:02.297308 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:02.298090 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:02.298094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:02.298094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:02.298099 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:02.298143 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:02.298154 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:02.298236 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:02.298349 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 12:04:02.298486 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260619 12:04:02.300078 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:02.300531 21672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528: Bootstrap starting.
I20260619 12:04:02.301424 21672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:02.301920 21672 log.cc:824] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:02.302796 21672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528: No bootstrap required, opened a new log
I20260619 12:04:02.303645 21675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:02.302296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34547 (local address 127.0.0.1:42799)
0619 12:04:02.302373 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:02.302376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:02.302562 (+ 186us) server_negotiation.cc:407] Connection header received
0619 12:04:02.302642 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:02.302642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:02.302648 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:02.302707 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:02.303349 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:02.303350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:02.303351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:02.303354 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:02.303369 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:02.303373 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:02.303427 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:02.303494 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 12:04:02.303544 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:04:02.305508 21672 raft_consensus.cc:348] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57a7ec39d73414a8f09a0574ad4b528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260619 12:04:02.305678 21672 raft_consensus.cc:374] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:02.305703 21672 raft_consensus.cc:729] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57a7ec39d73414a8f09a0574ad4b528, State: Initialized, Role: FOLLOWER
I20260619 12:04:02.305863 21672 consensus_queue.cc:260] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57a7ec39d73414a8f09a0574ad4b528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260619 12:04:02.305949 21672 raft_consensus.cc:388] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:02.305992 21672 raft_consensus.cc:482] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:02.306047 21672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:02.306744 21672 raft_consensus.cc:504] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57a7ec39d73414a8f09a0574ad4b528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260619 12:04:02.306875 21672 leader_election.cc:302] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f57a7ec39d73414a8f09a0574ad4b528; no voters:
I20260619 12:04:02.307123 21672 leader_election.cc:288] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:02.307252 21682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:02.307420 21672 sys_catalog.cc:564] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:02.307516 21682 raft_consensus.cc:686] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [term 1 LEADER]: Becoming Leader. State: Replica: f57a7ec39d73414a8f09a0574ad4b528, State: Running, Role: LEADER
I20260619 12:04:02.307642 21682 consensus_queue.cc:237] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57a7ec39d73414a8f09a0574ad4b528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260619 12:04:02.308101 21682 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f57a7ec39d73414a8f09a0574ad4b528. Latest consensus state: current_term: 1 leader_uuid: "f57a7ec39d73414a8f09a0574ad4b528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57a7ec39d73414a8f09a0574ad4b528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } } }
I20260619 12:04:02.308189 21682 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:02.308141 21683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f57a7ec39d73414a8f09a0574ad4b528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57a7ec39d73414a8f09a0574ad4b528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } } }
I20260619 12:04:02.308249 21683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:02.309036 21691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:02.309655 21691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:02.311354 21691 catalog_manager.cc:1347] Generated new cluster ID: fae62dbb5edb44a49c97c269729207e4
I20260619 12:04:02.311412 21691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:02.325475 21691 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:02.325700 21691 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:02.332576 21691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f57a7ec39d73414a8f09a0574ad4b528: Generated new TSK 0
I20260619 12:04:02.332846 21691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:02.339502 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21604
2026-06-19T12:04:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:02 dist-test-slave-fbq8 krb5kdc[21703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:02 dist-test-slave-fbq8 krb5kdc[21703](info): set up 2 sockets
Jun 19 12:04:02 dist-test-slave-fbq8 krb5kdc[21703](info): commencing operation
krb5kdc: starting...
W20260619 12:04:03.401193 815 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:03 dist-test-slave-fbq8 krb5kdc[21703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:03.544183 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:03.629062 21716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:03.629298 21716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:03.629323 21716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:03.630630 21716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:03.630692 21716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:03.630707 21716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:03.630719 21716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:03.630731 21716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:03.632222 21716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:03.632440 21716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:03.632625 21716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:03.638216 21722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:03.638259 21721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:03.638276 21716 server_base.cc:1034] running on GCE node
W20260619 12:04:03.638232 21724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:03.638954 21716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:03.639350 21716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:03.640507 21716 hybrid_clock.cc:648] HybridClock initialized: now 1781870643640490 us; error 71 us; skew 500 ppm
Jun 19 12:04:03 dist-test-slave-fbq8 krb5kdc[21703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:03.643404 21716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:03.643867 21716 webserver.cc:460] Webserver started at http://127.0.0.1:45931/ using document root <none> and password file <none>
I20260619 12:04:03.644133 21716 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:03.644204 21716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:03.644371 21716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:03.645639 21716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "33380dffe8f2436c92ef2d54b7df7a13"
format_stamp: "Formatted at 2026-06-19 12:04:03 on dist-test-slave-fbq8"
server_key: "fd5c49856b28fc8e358156b5d89ef021"
server_key_iv: "37623446d370386afbacc1fd354afbbf"
server_key_version: "encryptionkey@0"
I20260619 12:04:03.646037 21716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "33380dffe8f2436c92ef2d54b7df7a13"
format_stamp: "Formatted at 2026-06-19 12:04:03 on dist-test-slave-fbq8"
server_key: "fd5c49856b28fc8e358156b5d89ef021"
server_key_iv: "37623446d370386afbacc1fd354afbbf"
server_key_version: "encryptionkey@0"
I20260619 12:04:03.647620 21716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:04:03.648572 21731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:03.648911 21716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:04:03.649029 21716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "33380dffe8f2436c92ef2d54b7df7a13"
format_stamp: "Formatted at 2026-06-19 12:04:03 on dist-test-slave-fbq8"
server_key: "fd5c49856b28fc8e358156b5d89ef021"
server_key_iv: "37623446d370386afbacc1fd354afbbf"
server_key_version: "encryptionkey@0"
I20260619 12:04:03.649109 21716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:03.667207 21716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:03.669739 21716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:03.669924 21716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:03.674818 21716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45909
I20260619 12:04:03.674932 21783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45909 every 8 connection(s)
I20260619 12:04:03.675283 21716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:03.676311 21784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:03.678876 21784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13: Bootstrap starting.
I20260619 12:04:03.679452 21784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:03.679699 21784 log.cc:824] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:03.680487 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21716
I20260619 12:04:03.680629 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:03.680624 21784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13: No bootstrap required, opened a new log
I20260619 12:04:03.680913 815 external_mini_cluster.cc:1428] Setting key d77663af4102d6a41fab7c9ff2b4da0b
I20260619 12:04:03.683542 21784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33380dffe8f2436c92ef2d54b7df7a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260619 12:04:03.683729 21784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:03.683768 21784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33380dffe8f2436c92ef2d54b7df7a13, State: Initialized, Role: FOLLOWER
I20260619 12:04:03.683959 21784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33380dffe8f2436c92ef2d54b7df7a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260619 12:04:03.684060 21784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:03.684103 21784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:03.684152 21784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:03.684906 21784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33380dffe8f2436c92ef2d54b7df7a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260619 12:04:03.685034 21784 leader_election.cc:302] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33380dffe8f2436c92ef2d54b7df7a13; no voters:
I20260619 12:04:03.685245 21784 leader_election.cc:288] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:03.685336 21789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:03.685602 21789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [term 1 LEADER]: Becoming Leader. State: Replica: 33380dffe8f2436c92ef2d54b7df7a13, State: Running, Role: LEADER
I20260619 12:04:03.685621 21784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:03.685771 21789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33380dffe8f2436c92ef2d54b7df7a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260619 12:04:03.686254 21789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33380dffe8f2436c92ef2d54b7df7a13. Latest consensus state: current_term: 1 leader_uuid: "33380dffe8f2436c92ef2d54b7df7a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33380dffe8f2436c92ef2d54b7df7a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260619 12:04:03.686395 21789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:03.686647 21798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:03.687247 21790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33380dffe8f2436c92ef2d54b7df7a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33380dffe8f2436c92ef2d54b7df7a13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260619 12:04:03.687347 21790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:03.687618 21798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 12:04:03 dist-test-slave-fbq8 krb5kdc[21703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:04:03.690187 21798 catalog_manager.cc:1347] Generated new cluster ID: c8174d02faaf4cee913d9756bb3c35a4
I20260619 12:04:03.690255 21798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:03.694587 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:03.681550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:39419 (local address 127.0.0.1:45909)
0619 12:04:03.681734 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:03.681742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:03.681771 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:04:03.682286 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:03.682295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:03.682348 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:03.682486 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:03.683318 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:03.684767 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:03.686040 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:03.686748 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:03.686876 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:03.689798 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:03.689802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:03.689802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:03.689816 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:03.692122 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:03.692753 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:03.692757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:03.692757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:03.692870 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:03.693135 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:03.693136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:03.693136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:03.693473 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:03.693645 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:03.694207 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 12:04:03.694337 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260619 12:04:03.695847 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:03.702531 21798 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:03.703184 21798 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:03.705500 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:03.697502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:44379 (local address 127.0.0.1:45909)
0619 12:04:03.697573 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:03.697576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:03.697588 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:04:03.697614 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:03.697615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:03.697628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:03.697693 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:03.698344 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:03.699125 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:03.700206 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:03.700552 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:03.700633 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:03.702297 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:03.702298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:03.702298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:03.702305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:03.704118 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:03.704594 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:03.704595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:03.704595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:03.704692 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:03.704924 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:03.704925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:03.704925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:03.705061 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:03.705178 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:03.705332 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 12:04:03.705400 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:04:03.707820 21798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33380dffe8f2436c92ef2d54b7df7a13: Generated new TSK 0
I20260619 12:04:03.708038 21798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:03.716220 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21716
W20260619 12:04:03.722033 21701 connection.cc:535] client connection to 127.0.0.1:45909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45909): Connection reset by peer (error 104)
2026-06-19T12:04:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:03 dist-test-slave-fbq8 krb5kdc[21815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:03 dist-test-slave-fbq8 krb5kdc[21815](info): set up 2 sockets
Jun 19 12:04:03 dist-test-slave-fbq8 krb5kdc[21815](info): commencing operation
krb5kdc: starting...
W20260619 12:04:04.780992 815 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:04 dist-test-slave-fbq8 krb5kdc[21815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:04.916239 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:04.999253 21828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:04.999548 21828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:04.999577 21828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:05.001519 21828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:05.001595 21828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:05.001612 21828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:05.001626 21828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:05.001637 21828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:05.003747 21828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:05.004096 21828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:05.004361 21828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:05.010448 21836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:05.010771 21833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:05.010439 21834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:05.010978 21828 server_base.cc:1034] running on GCE node
I20260619 12:04:05.011304 21828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:05.011766 21828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:05.012980 21828 hybrid_clock.cc:648] HybridClock initialized: now 1781870645012951 us; error 54 us; skew 500 ppm
Jun 19 12:04:05 dist-test-slave-fbq8 krb5kdc[21815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:05.016400 21828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:05.017122 21828 webserver.cc:460] Webserver started at http://127.0.0.1:32865/ using document root <none> and password file <none>
I20260619 12:04:05.017379 21828 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:05.017432 21828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:05.017566 21828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:05.018774 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "5244172e773d4e41b1cd9a4fdb19a2e1"
format_stamp: "Formatted at 2026-06-19 12:04:05 on dist-test-slave-fbq8"
server_key: "65d14e9d29794e910a9156fe3469d189"
server_key_iv: "8650fbdf0a8f24ed0bb5bc4d1fe491ce"
server_key_version: "encryptionkey@0"
I20260619 12:04:05.019114 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "5244172e773d4e41b1cd9a4fdb19a2e1"
format_stamp: "Formatted at 2026-06-19 12:04:05 on dist-test-slave-fbq8"
server_key: "65d14e9d29794e910a9156fe3469d189"
server_key_iv: "8650fbdf0a8f24ed0bb5bc4d1fe491ce"
server_key_version: "encryptionkey@0"
I20260619 12:04:05.020836 21828 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:04:05.021848 21843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:05.022147 21828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:05.022267 21828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "5244172e773d4e41b1cd9a4fdb19a2e1"
format_stamp: "Formatted at 2026-06-19 12:04:05 on dist-test-slave-fbq8"
server_key: "65d14e9d29794e910a9156fe3469d189"
server_key_iv: "8650fbdf0a8f24ed0bb5bc4d1fe491ce"
server_key_version: "encryptionkey@0"
I20260619 12:04:05.022352 21828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:05.050074 21828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:05.053375 21828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:05.053550 21828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:05.058868 21828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42085
I20260619 12:04:05.058990 21895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42085 every 8 connection(s)
I20260619 12:04:05.059358 21828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:05.060575 21896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:05.062597 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21828
I20260619 12:04:05.062709 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:05.063071 815 external_mini_cluster.cc:1428] Setting key 4ffb64b7035364bb20bb7cd41e43fba3
I20260619 12:04:05.064061 21896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1: Bootstrap starting.
I20260619 12:04:05.064906 21896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:05.065254 21896 log.cc:824] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:05.066391 21896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1: No bootstrap required, opened a new log
I20260619 12:04:05.068431 21896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42085 } }
I20260619 12:04:05.068674 21896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:05.068754 21896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5244172e773d4e41b1cd9a4fdb19a2e1, State: Initialized, Role: FOLLOWER
I20260619 12:04:05.068914 21896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42085 } }
I20260619 12:04:05.068992 21896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:05.069015 21896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:05.069095 21896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:05.069850 21896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42085 } }
I20260619 12:04:05.069998 21896 leader_election.cc:302] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5244172e773d4e41b1cd9a4fdb19a2e1; no voters:
I20260619 12:04:05.070245 21896 leader_election.cc:288] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:05.070290 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:05.064043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:50439 (local address 127.0.0.1:42085)
0619 12:04:05.064315 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:05.064323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:05.064351 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:04:05.065115 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:05.065123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:05.065182 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:05.065312 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:05.065647 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:05.066717 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:05.067919 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:05.068384 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:05.068526 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:05.069415 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:05.069419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:05.069420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:05.069633 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:05.069671 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:05.069684 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:05.069781 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:05.069867 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 12:04:05.069900 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":160,"threads_started":1}
I20260619 12:04:05.070325 21901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:05.070598 21901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [term 1 LEADER]: Becoming Leader. State: Replica: 5244172e773d4e41b1cd9a4fdb19a2e1, State: Running, Role: LEADER
I20260619 12:04:05.070765 21896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:05.070747 21901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42085 } }
I20260619 12:04:05.071245 21903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5244172e773d4e41b1cd9a4fdb19a2e1. Latest consensus state: current_term: 1 leader_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42085 } } }
I20260619 12:04:05.071358 21903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:05.071573 21902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244172e773d4e41b1cd9a4fdb19a2e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42085 } } }
I20260619 12:04:05.071770 21902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:05.071890 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 12:04:05.073560 21916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:05.073688 21916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:05.073824 21906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:05.074149 21906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:05.076767 21906 catalog_manager.cc:1347] Generated new cluster ID: 188c043c55fd4bfe836afe65589baf6a
I20260619 12:04:05.076859 21906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:05.079241 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:05.073750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:44201 (local address 127.0.0.1:42085)
0619 12:04:05.073796 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:05.073798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:05.075176 (+ 1378us) server_negotiation.cc:407] Connection header received
0619 12:04:05.075260 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:05.075261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:05.075276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:05.075357 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:05.075637 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:05.076350 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:05.077523 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:05.077787 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:05.078019 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:05.078956 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:05.078958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:05.078958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:05.078963 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:05.078980 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:05.078985 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:05.079047 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:05.079124 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 12:04:05.079135 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 12:04:05.128360 21906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:05.129459 21906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:05.138311 21906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5244172e773d4e41b1cd9a4fdb19a2e1: Generated new TSK 0
I20260619 12:04:05.138569 21906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:05.153441 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21828
W20260619 12:04:05.158830 21813 connection.cc:535] client connection to 127.0.0.1:42085 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42085): Connection reset by peer (error 104)
2026-06-19T12:04:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:05 dist-test-slave-fbq8 krb5kdc[21927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:05 dist-test-slave-fbq8 krb5kdc[21927](info): set up 2 sockets
Jun 19 12:04:05 dist-test-slave-fbq8 krb5kdc[21927](info): commencing operation
krb5kdc: starting...
W20260619 12:04:06.249122 815 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:06 dist-test-slave-fbq8 krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:06.388895 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:06.482460 21940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:06.482719 21940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:06.482744 21940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:06.483886 21940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:06.483925 21940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:06.483939 21940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:06.483952 21940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:06.483964 21940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:06.485607 21940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:06.485841 21940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:06.486037 21940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:06.491654 21948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:06.491671 21946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:06.491649 21945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:06.491889 21940 server_base.cc:1034] running on GCE node
I20260619 12:04:06.492408 21940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:06.492816 21940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:06.494035 21940 hybrid_clock.cc:648] HybridClock initialized: now 1781870646494026 us; error 47 us; skew 500 ppm
Jun 19 12:04:06 dist-test-slave-fbq8 krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:06.501516 21940 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:06.502030 21940 webserver.cc:460] Webserver started at http://127.0.0.1:38229/ using document root <none> and password file <none>
I20260619 12:04:06.502275 21940 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:06.502346 21940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:06.502509 21940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:06.503672 21940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9"
format_stamp: "Formatted at 2026-06-19 12:04:06 on dist-test-slave-fbq8"
server_key: "6e9b4416d92fddf22843a12903482f9c"
server_key_iv: "b6fd004f42fdd9e3712791d7ed15feff"
server_key_version: "encryptionkey@0"
I20260619 12:04:06.504065 21940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9"
format_stamp: "Formatted at 2026-06-19 12:04:06 on dist-test-slave-fbq8"
server_key: "6e9b4416d92fddf22843a12903482f9c"
server_key_iv: "b6fd004f42fdd9e3712791d7ed15feff"
server_key_version: "encryptionkey@0"
I20260619 12:04:06.505764 21940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 12:04:06.506763 21955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:06.507052 21940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:06.507146 21940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9"
format_stamp: "Formatted at 2026-06-19 12:04:06 on dist-test-slave-fbq8"
server_key: "6e9b4416d92fddf22843a12903482f9c"
server_key_iv: "b6fd004f42fdd9e3712791d7ed15feff"
server_key_version: "encryptionkey@0"
I20260619 12:04:06.507238 21940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:06.510753 21940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:06.513132 21940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:06.513248 21940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:06.517793 21940 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35041
I20260619 12:04:06.517938 22007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35041 every 8 connection(s)
I20260619 12:04:06.518221 21940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:06.519316 22008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:06.522200 22008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9: Bootstrap starting.
I20260619 12:04:06.522925 22008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:06.523255 22008 log.cc:824] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:06.524832 22008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9: No bootstrap required, opened a new log
I20260619 12:04:06.524991 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 21940
I20260619 12:04:06.525172 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:06.525391 815 external_mini_cluster.cc:1428] Setting key 44b16e3cf305f7d802698b03296205b6
I20260619 12:04:06.529392 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:06.526218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:48443 (local address 127.0.0.1:35041)
0619 12:04:06.526518 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:06.526527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:06.526560 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:04:06.527078 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:06.527086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:06.527113 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:06.527326 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:06.528511 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:06.528515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:06.528515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:06.528520 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:06.528575 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:06.528588 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:06.528773 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:06.528878 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 12:04:06.528946 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":180,"threads_started":1}
I20260619 12:04:06.529320 22008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260619 12:04:06.529726 22008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:06.529794 22008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2fcd7a4e9d42369dffc5fd13878fc9, State: Initialized, Role: FOLLOWER
I20260619 12:04:06.529981 22008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260619 12:04:06.530100 22008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:06.530150 22008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:06.530215 22008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:06.531037 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:06.531302 22008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260619 12:04:06.531435 22008 leader_election.cc:302] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b2fcd7a4e9d42369dffc5fd13878fc9; no voters:
I20260619 12:04:06.531680 22008 leader_election.cc:288] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:06.531776 22013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:06.532006 22008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:06.532058 22013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 6b2fcd7a4e9d42369dffc5fd13878fc9, State: Running, Role: LEADER
I20260619 12:04:06.532215 22013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260619 12:04:06.532714 22015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2fcd7a4e9d42369dffc5fd13878fc9. Latest consensus state: current_term: 1 leader_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } } }
I20260619 12:04:06.532829 22015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:06.532689 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2fcd7a4e9d42369dffc5fd13878fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } } }
I20260619 12:04:06.532917 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:06.533115 22021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:06.533869 22021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:06.535938 22021 catalog_manager.cc:1347] Generated new cluster ID: 7eb39e5f4c884fbcb218e38f26a6d31b
I20260619 12:04:06.536079 22021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:06.538070 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:06.533365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:56127 (local address 127.0.0.1:35041)
0619 12:04:06.533410 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:06.533412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:06.534178 (+ 766us) server_negotiation.cc:407] Connection header received
0619 12:04:06.536849 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:06.536851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:06.536859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:06.536951 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:06.537772 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:06.537774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:06.537775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:06.537780 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:06.537804 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:06.537810 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:06.537876 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:06.537981 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 12:04:06.537997 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 12:04:06.551348 22021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:06.551755 22021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:06.558701 22021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b2fcd7a4e9d42369dffc5fd13878fc9: Generated new TSK 0
I20260619 12:04:06.558959 22021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:06.565151 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 21940
2026-06-19T12:04:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:06 dist-test-slave-fbq8 krb5kdc[22039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:06 dist-test-slave-fbq8 krb5kdc[22039](info): set up 2 sockets
Jun 19 12:04:06 dist-test-slave-fbq8 krb5kdc[22039](info): commencing operation
krb5kdc: starting...
W20260619 12:04:07.612910 815 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:07 dist-test-slave-fbq8 krb5kdc[22039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:07.747202 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:07.840418 22052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:07.840758 22052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:07.840799 22052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:07.842064 22052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:07.842109 22052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:07.842124 22052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:07.842139 22052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:07.842151 22052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:07.843695 22052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:07.843993 22052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:07.844247 22052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:07.849676 22057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:07.849697 22060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:07.849697 22058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:07.850224 22052 server_base.cc:1034] running on GCE node
I20260619 12:04:07.850579 22052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:07.850891 22052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:07.852070 22052 hybrid_clock.cc:648] HybridClock initialized: now 1781870647852031 us; error 58 us; skew 500 ppm
Jun 19 12:04:07 dist-test-slave-fbq8 krb5kdc[22039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:07.855158 22052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:07.855695 22052 webserver.cc:460] Webserver started at http://127.0.0.1:34221/ using document root <none> and password file <none>
I20260619 12:04:07.855947 22052 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:07.856000 22052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:07.856184 22052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:07.857389 22052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "91c0336ebeed4f11a1fc45cb7a40fb25"
format_stamp: "Formatted at 2026-06-19 12:04:07 on dist-test-slave-fbq8"
server_key: "1ea11e259bee3014fbdf657bb5a34ec4"
server_key_iv: "1bcec3a3aa4a05a320e3dbc917e5aa34"
server_key_version: "encryptionkey@0"
I20260619 12:04:07.857782 22052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "91c0336ebeed4f11a1fc45cb7a40fb25"
format_stamp: "Formatted at 2026-06-19 12:04:07 on dist-test-slave-fbq8"
server_key: "1ea11e259bee3014fbdf657bb5a34ec4"
server_key_iv: "1bcec3a3aa4a05a320e3dbc917e5aa34"
server_key_version: "encryptionkey@0"
I20260619 12:04:07.859195 22052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:04:07.860234 22067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:07.860510 22052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:07.860633 22052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "91c0336ebeed4f11a1fc45cb7a40fb25"
format_stamp: "Formatted at 2026-06-19 12:04:07 on dist-test-slave-fbq8"
server_key: "1ea11e259bee3014fbdf657bb5a34ec4"
server_key_iv: "1bcec3a3aa4a05a320e3dbc917e5aa34"
server_key_version: "encryptionkey@0"
I20260619 12:04:07.860776 22052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:07.876164 22052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:07.878679 22052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:07.878871 22052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:07.883531 22052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32895
I20260619 12:04:07.883623 22119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32895 every 8 connection(s)
I20260619 12:04:07.884040 22052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:07.885257 22120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:07.887955 22120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25: Bootstrap starting.
I20260619 12:04:07.888628 22120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:07.889020 22120 log.cc:824] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:07.889921 22120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25: No bootstrap required, opened a new log
I20260619 12:04:07.891644 22120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32895 } }
I20260619 12:04:07.891804 22120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:07.891829 22120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c0336ebeed4f11a1fc45cb7a40fb25, State: Initialized, Role: FOLLOWER
I20260619 12:04:07.891990 22120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32895 } }
I20260619 12:04:07.892078 22120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:07.892102 22120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:07.892172 22120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:07.892838 22120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32895 } }
I20260619 12:04:07.892942 22120 leader_election.cc:302] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91c0336ebeed4f11a1fc45cb7a40fb25; no voters:
I20260619 12:04:07.893216 22120 leader_election.cc:288] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:07.893237 22123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:07.893245 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22052
I20260619 12:04:07.893388 22123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [term 1 LEADER]: Becoming Leader. State: Replica: 91c0336ebeed4f11a1fc45cb7a40fb25, State: Running, Role: LEADER
I20260619 12:04:07.893414 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:07.893479 22123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32895 } }
I20260619 12:04:07.893545 22120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:07.893615 815 external_mini_cluster.cc:1428] Setting key 348b340fb1c41a3ed1f54f519f8964ee
I20260619 12:04:07.894032 22124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32895 } } }
I20260619 12:04:07.894177 22124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:07.894428 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c0336ebeed4f11a1fc45cb7a40fb25. Latest consensus state: current_term: 1 leader_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c0336ebeed4f11a1fc45cb7a40fb25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32895 } } }
I20260619 12:04:07.894531 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25 [sys.catalog]: This master's current role is: LEADER
W20260619 12:04:07.895644 22140 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:07.895704 22140 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:07.895767 22133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:07.895967 22133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:07.897868 22133 catalog_manager.cc:1347] Generated new cluster ID: 69f3ea7e1faf420ba0cf9d536122ef2a
I20260619 12:04:07.897923 22133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:04:07 dist-test-slave-fbq8 krb5kdc[22039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:04:07.909162 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:07.894282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:39305 (local address 127.0.0.1:32895)
0619 12:04:07.894512 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:07.894523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:07.894897 (+ 374us) server_negotiation.cc:407] Connection header received
0619 12:04:07.895202 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:07.895212 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:07.895311 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:07.895571 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:07.896814 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:07.898393 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:07.899567 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:07.900251 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:07.900382 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:07.903890 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:07.903895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:07.903896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:07.903911 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:07.906722 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:07.907354 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:07.907359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:07.907359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:07.907500 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:07.907795 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:07.907796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:07.907796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:07.908340 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:07.908516 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:07.908752 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 12:04:07.908880 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260619 12:04:07.910801 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:07.921211 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:07.912537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:49427 (local address 127.0.0.1:32895)
0619 12:04:07.912598 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:07.912600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:07.912895 (+ 295us) server_negotiation.cc:407] Connection header received
0619 12:04:07.912958 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:07.912959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:07.912972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:07.913037 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:07.913778 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:07.914586 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:07.915769 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:07.916093 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:07.916156 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:07.917853 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:07.917854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:07.917855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:07.917862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:07.919735 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:07.920238 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:07.920239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:07.920239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:07.920298 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:07.920559 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:07.920559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:07.920560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:07.920740 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:07.920862 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:07.921029 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:04:07.921106 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:04:07.926234 22133 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:07.927064 22133 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:07.956218 22133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91c0336ebeed4f11a1fc45cb7a40fb25: Generated new TSK 0
I20260619 12:04:07.956496 22133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:07.960554 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22052
W20260619 12:04:07.966008 22037 connection.cc:535] client connection to 127.0.0.1:32895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32895): Connection reset by peer (error 104)
2026-06-19T12:04:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:08 dist-test-slave-fbq8 krb5kdc[22151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:08 dist-test-slave-fbq8 krb5kdc[22151](info): set up 2 sockets
Jun 19 12:04:08 dist-test-slave-fbq8 krb5kdc[22151](info): commencing operation
krb5kdc: starting...
W20260619 12:04:09.040925 815 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:09 dist-test-slave-fbq8 krb5kdc[22151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:09.172997 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:09.263988 22164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:09.264297 22164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:09.264326 22164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:09.265563 22164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:09.265615 22164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:09.265630 22164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:09.265643 22164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:09.265656 22164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:09.267155 22164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:09.267376 22164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:09.267555 22164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:09.273103 22169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:09.273123 22170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:09.273128 22172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:09.273576 22164 server_base.cc:1034] running on GCE node
I20260619 12:04:09.273901 22164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:09.274309 22164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:09.275525 22164 hybrid_clock.cc:648] HybridClock initialized: now 1781870649275480 us; error 66 us; skew 500 ppm
Jun 19 12:04:09 dist-test-slave-fbq8 krb5kdc[22151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:09.278656 22164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:09.279184 22164 webserver.cc:460] Webserver started at http://127.0.0.1:38265/ using document root <none> and password file <none>
I20260619 12:04:09.279470 22164 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:09.279543 22164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:09.279686 22164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:09.280859 22164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "c72371b0211b463eabca82e17dcbd120"
format_stamp: "Formatted at 2026-06-19 12:04:09 on dist-test-slave-fbq8"
server_key: "467ab28914cdb1833425a0acb5033500"
server_key_iv: "84a9753688677bdd1bb4dd8c612c69fb"
server_key_version: "encryptionkey@0"
I20260619 12:04:09.281220 22164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "c72371b0211b463eabca82e17dcbd120"
format_stamp: "Formatted at 2026-06-19 12:04:09 on dist-test-slave-fbq8"
server_key: "467ab28914cdb1833425a0acb5033500"
server_key_iv: "84a9753688677bdd1bb4dd8c612c69fb"
server_key_version: "encryptionkey@0"
I20260619 12:04:09.282783 22164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:04:09.284065 22179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:09.284411 22164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:09.284552 22164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "c72371b0211b463eabca82e17dcbd120"
format_stamp: "Formatted at 2026-06-19 12:04:09 on dist-test-slave-fbq8"
server_key: "467ab28914cdb1833425a0acb5033500"
server_key_iv: "84a9753688677bdd1bb4dd8c612c69fb"
server_key_version: "encryptionkey@0"
I20260619 12:04:09.284673 22164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:09.307286 22164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:09.310016 22164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:09.310206 22164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:09.315502 22164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42191
I20260619 12:04:09.315577 22231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42191 every 8 connection(s)
I20260619 12:04:09.315999 22164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:09.317178 22232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:09.318768 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22164
I20260619 12:04:09.319005 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:09.319244 815 external_mini_cluster.cc:1428] Setting key 6c5098a33ee79ba91e0f8a869f291f2a
I20260619 12:04:09.319923 22232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120: Bootstrap starting.
I20260619 12:04:09.320775 22232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:09.321118 22232 log.cc:824] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:09.322131 22232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120: No bootstrap required, opened a new log
I20260619 12:04:09.324342 22232 raft_consensus.cc:348] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72371b0211b463eabca82e17dcbd120" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42191 } }
I20260619 12:04:09.324522 22232 raft_consensus.cc:374] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:09.324568 22232 raft_consensus.cc:729] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72371b0211b463eabca82e17dcbd120, State: Initialized, Role: FOLLOWER
I20260619 12:04:09.324744 22232 consensus_queue.cc:260] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72371b0211b463eabca82e17dcbd120" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42191 } }
I20260619 12:04:09.324849 22232 raft_consensus.cc:388] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:09.324877 22232 raft_consensus.cc:482] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:09.324955 22232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:09.325760 22232 raft_consensus.cc:504] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72371b0211b463eabca82e17dcbd120" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42191 } }
I20260619 12:04:09.325877 22232 leader_election.cc:302] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c72371b0211b463eabca82e17dcbd120; no voters:
I20260619 12:04:09.326067 22232 leader_election.cc:288] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:09.326143 22237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:09.326342 22237 raft_consensus.cc:686] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [term 1 LEADER]: Becoming Leader. State: Replica: c72371b0211b463eabca82e17dcbd120, State: Running, Role: LEADER
I20260619 12:04:09.326422 22232 sys_catalog.cc:564] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:09.326519 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:09.319962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:38513 (local address 127.0.0.1:42191)
0619 12:04:09.320207 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:09.320217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:09.320254 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:04:09.321005 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:09.321015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:09.321090 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:09.321274 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:09.321594 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:09.322989 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:09.323839 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:09.324481 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:09.324616 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:09.325573 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:09.325577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:09.325578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:09.325801 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:09.325844 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:09.325859 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:09.325970 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:09.326102 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 12:04:09.326278 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":122,"threads_started":1}
I20260619 12:04:09.326574 22237 consensus_queue.cc:237] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72371b0211b463eabca82e17dcbd120" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42191 } }
I20260619 12:04:09.327009 22237 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c72371b0211b463eabca82e17dcbd120. Latest consensus state: current_term: 1 leader_uuid: "c72371b0211b463eabca82e17dcbd120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72371b0211b463eabca82e17dcbd120" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42191 } } }
I20260619 12:04:09.327232 22237 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:09.327157 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c72371b0211b463eabca82e17dcbd120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72371b0211b463eabca82e17dcbd120" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42191 } } }
I20260619 12:04:09.327498 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:09.327745 22246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:09.327845 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:09.328495 22246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:09.330718 22246 catalog_manager.cc:1347] Generated new cluster ID: 3cc906a769cb4f13bf1b22ea8d6569e5
I20260619 12:04:09.330792 22246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:09.334134 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:09.329609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:41521 (local address 127.0.0.1:42191)
0619 12:04:09.329647 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:09.329649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:09.330095 (+ 446us) server_negotiation.cc:407] Connection header received
0619 12:04:09.330182 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:09.330183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:09.330195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:09.330262 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:09.330459 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:09.331253 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:09.332356 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:09.332704 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:09.332760 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:09.333750 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:09.333751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:09.333752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:09.333757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:09.333775 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:09.333779 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:09.333826 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:09.334030 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 12:04:09.334042 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 12:04:09.355960 22246 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:09.356916 22246 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:09.360915 22246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c72371b0211b463eabca82e17dcbd120: Generated new TSK 0
I20260619 12:04:09.361188 22246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:09.364804 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22164
W20260619 12:04:09.370214 22149 connection.cc:535] client connection to 127.0.0.1:42191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42191): Connection reset by peer (error 104)
2026-06-19T12:04:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:09 dist-test-slave-fbq8 krb5kdc[22263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:09 dist-test-slave-fbq8 krb5kdc[22263](info): set up 2 sockets
Jun 19 12:04:09 dist-test-slave-fbq8 krb5kdc[22263](info): commencing operation
krb5kdc: starting...
W20260619 12:04:10.428864 815 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:10 dist-test-slave-fbq8 krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:04:10.564616 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:10.648180 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:10.648447 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:10.648486 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:10.649718 22276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:10.649802 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:10.649830 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:10.649863 22276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:10.649891 22276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:10.651409 22276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:10.651705 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:10.651947 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:10.657657 22284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:10.657657 22281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:10.657733 22282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:10.658087 22276 server_base.cc:1034] running on GCE node
I20260619 12:04:10.658473 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:10.658816 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:10.660066 22276 hybrid_clock.cc:648] HybridClock initialized: now 1781870650660002 us; error 58 us; skew 500 ppm
Jun 19 12:04:10 dist-test-slave-fbq8 krb5kdc[22263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:10.662931 22276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:04:10.663488 22276 webserver.cc:460] Webserver started at http://127.0.0.1:34421/ using document root <none> and password file <none>
I20260619 12:04:10.663762 22276 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:10.663834 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:10.663981 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:10.665156 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09"
format_stamp: "Formatted at 2026-06-19 12:04:10 on dist-test-slave-fbq8"
server_key: "3e3922b9b96cefa4ff45d2215444642a"
server_key_iv: "3944e1e4692ec31f275bb44e7269520f"
server_key_version: "encryptionkey@0"
I20260619 12:04:10.665526 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09"
format_stamp: "Formatted at 2026-06-19 12:04:10 on dist-test-slave-fbq8"
server_key: "3e3922b9b96cefa4ff45d2215444642a"
server_key_iv: "3944e1e4692ec31f275bb44e7269520f"
server_key_version: "encryptionkey@0"
I20260619 12:04:10.666963 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 12:04:10.667879 22291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:10.668179 22276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:10.668298 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09"
format_stamp: "Formatted at 2026-06-19 12:04:10 on dist-test-slave-fbq8"
server_key: "3e3922b9b96cefa4ff45d2215444642a"
server_key_iv: "3944e1e4692ec31f275bb44e7269520f"
server_key_version: "encryptionkey@0"
I20260619 12:04:10.668404 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:10.671932 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:10.674396 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:10.674577 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:10.679272 22276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46063
I20260619 12:04:10.679339 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46063 every 8 connection(s)
I20260619 12:04:10.679700 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:10.680439 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22276
I20260619 12:04:10.680541 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:10.680761 815 external_mini_cluster.cc:1428] Setting key 141308939346c58ed56ff80b7e6e4e00
I20260619 12:04:10.681296 22344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:10.683617 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:10.681308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:39631 (local address 127.0.0.1:46063)
0619 12:04:10.681519 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:10.681527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:10.681552 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:04:10.681983 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:10.681989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:10.682012 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:10.682127 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:10.683123 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:10.683126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:10.683126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:10.683130 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:10.683170 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:10.683181 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:10.683244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:10.683392 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 12:04:10.683449 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
I20260619 12:04:10.684989 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:10.685966 22344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09: Bootstrap starting.
I20260619 12:04:10.686885 22344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:10.687222 22344 log.cc:824] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:10.688169 22344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09: No bootstrap required, opened a new log
I20260619 12:04:10.688488 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:10.686948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:53163 (local address 127.0.0.1:46063)
0619 12:04:10.687000 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:10.687002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:10.687162 (+ 160us) server_negotiation.cc:407] Connection header received
0619 12:04:10.687251 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:10.687252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:10.687258 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:10.687314 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:10.688160 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:10.688161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:10.688161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:10.688166 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:10.688182 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:10.688188 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:10.688263 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:10.688408 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 12:04:10.688424 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 12:04:10.690625 22344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260619 12:04:10.690790 22344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:10.690817 22344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d133fcbe134f7bb6e1a4ed0cc73b09, State: Initialized, Role: FOLLOWER
I20260619 12:04:10.690961 22344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260619 12:04:10.691064 22344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:10.691090 22344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:10.691123 22344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:10.691762 22344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260619 12:04:10.691864 22344 leader_election.cc:302] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d133fcbe134f7bb6e1a4ed0cc73b09; no voters:
I20260619 12:04:10.692019 22344 leader_election.cc:288] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:10.692113 22354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:10.692334 22344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:10.692348 22354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [term 1 LEADER]: Becoming Leader. State: Replica: 64d133fcbe134f7bb6e1a4ed0cc73b09, State: Running, Role: LEADER
I20260619 12:04:10.692585 22354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } }
I20260619 12:04:10.693125 22355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } } }
I20260619 12:04:10.693461 22355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:10.693192 22356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d133fcbe134f7bb6e1a4ed0cc73b09. Latest consensus state: current_term: 1 leader_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d133fcbe134f7bb6e1a4ed0cc73b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46063 } } }
I20260619 12:04:10.693747 22356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:10.693914 22363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:10.694484 22363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:10.696287 22363 catalog_manager.cc:1347] Generated new cluster ID: 76ca6101afad48a7b1abe9d3439bfdfa
I20260619 12:04:10.696358 22363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:10.713976 22363 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:10.714262 22363 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:10.717572 22363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d133fcbe134f7bb6e1a4ed0cc73b09: Generated new TSK 0
I20260619 12:04:10.717828 22363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:10.721184 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22276
2026-06-19T12:04:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:10 dist-test-slave-fbq8 krb5kdc[22375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:10 dist-test-slave-fbq8 krb5kdc[22375](info): set up 2 sockets
Jun 19 12:04:10 dist-test-slave-fbq8 krb5kdc[22375](info): commencing operation
krb5kdc: starting...
W20260619 12:04:11.769230 815 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:11 dist-test-slave-fbq8 krb5kdc[22375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:11.903703 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36869
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:36869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:11.996526 22388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:11.996940 22388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:11.996985 22388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:11.999091 22388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:11.999267 22388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:11.999308 22388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:11.999358 22388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:11.999388 22388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:12.003284 22388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:36869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:12.003657 22388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:12.003935 22388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:12.009972 22394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:12.009995 22396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:12.010349 22388 server_base.cc:1034] running on GCE node
W20260619 12:04:12.009987 22393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:12.010766 22388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:12.011128 22388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:12.012318 22388 hybrid_clock.cc:648] HybridClock initialized: now 1781870652012275 us; error 72 us; skew 500 ppm
Jun 19 12:04:12 dist-test-slave-fbq8 krb5kdc[22375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:12.019723 22388 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:12.020300 22388 webserver.cc:460] Webserver started at http://127.0.203.254:44391/ using document root <none> and password file <none>
I20260619 12:04:12.020581 22388 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:12.020670 22388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:12.020833 22388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:12.022029 22388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "006bd82992a241c59125d80646d21e1c"
format_stamp: "Formatted at 2026-06-19 12:04:12 on dist-test-slave-fbq8"
server_key: "9680599fdbf81c3447db82d9d8a58dab"
server_key_iv: "77c2414e871d9ee9d72ddb6c6102baed"
server_key_version: "encryptionkey@0"
I20260619 12:04:12.022415 22388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "006bd82992a241c59125d80646d21e1c"
format_stamp: "Formatted at 2026-06-19 12:04:12 on dist-test-slave-fbq8"
server_key: "9680599fdbf81c3447db82d9d8a58dab"
server_key_iv: "77c2414e871d9ee9d72ddb6c6102baed"
server_key_version: "encryptionkey@0"
I20260619 12:04:12.024000 22388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:04:12.025051 22403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:12.025393 22388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:12.025535 22388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "006bd82992a241c59125d80646d21e1c"
format_stamp: "Formatted at 2026-06-19 12:04:12 on dist-test-slave-fbq8"
server_key: "9680599fdbf81c3447db82d9d8a58dab"
server_key_iv: "77c2414e871d9ee9d72ddb6c6102baed"
server_key_version: "encryptionkey@0"
I20260619 12:04:12.025628 22388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:12.035241 22388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:12.038213 22388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:12.038403 22388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:12.043023 22388 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:36869
I20260619 12:04:12.043152 22455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:36869 every 8 connection(s)
I20260619 12:04:12.043488 22388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:12.044551 22456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:12.047268 22456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c: Bootstrap starting.
I20260619 12:04:12.047946 22456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:12.048269 22456 log.cc:824] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:12.049299 22456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c: No bootstrap required, opened a new log
I20260619 12:04:12.050045 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22388
I20260619 12:04:12.050182 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:12.050395 815 external_mini_cluster.cc:1428] Setting key bcaa73b5f1d2361e6df1a8f3f28fa781
I20260619 12:04:12.051708 22456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006bd82992a241c59125d80646d21e1c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36869 } }
I20260619 12:04:12.051891 22456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:12.051930 22456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006bd82992a241c59125d80646d21e1c, State: Initialized, Role: FOLLOWER
I20260619 12:04:12.052126 22456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006bd82992a241c59125d80646d21e1c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36869 } }
I20260619 12:04:12.052224 22456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:12.052263 22456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:12.052318 22456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:12.053458 22456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006bd82992a241c59125d80646d21e1c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36869 } }
I20260619 12:04:12.053568 22456 leader_election.cc:302] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 006bd82992a241c59125d80646d21e1c; no voters:
I20260619 12:04:12.053836 22456 leader_election.cc:288] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:12.053857 22461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:12.054086 22461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [term 1 LEADER]: Becoming Leader. State: Replica: 006bd82992a241c59125d80646d21e1c, State: Running, Role: LEADER
I20260619 12:04:12.054283 22456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:12.054311 22461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006bd82992a241c59125d80646d21e1c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36869 } }
I20260619 12:04:12.054804 22461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "006bd82992a241c59125d80646d21e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006bd82992a241c59125d80646d21e1c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36869 } } }
I20260619 12:04:12.055306 22461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:12.055187 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 006bd82992a241c59125d80646d21e1c. Latest consensus state: current_term: 1 leader_uuid: "006bd82992a241c59125d80646d21e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006bd82992a241c59125d80646d21e1c" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36869 } } }
I20260619 12:04:12.055742 22470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:12.056525 22470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:12.055758 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:12.058634 22470 catalog_manager.cc:1347] Generated new cluster ID: aabd0d9393ee48e8a09fe05a1913568f
I20260619 12:04:12.058686 22470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:04:12 dist-test-slave-fbq8 krb5kdc[22375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:12.064178 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:12.051038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42221 (local address 127.0.203.254:36869)
0619 12:04:12.051258 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:12.051268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:12.051301 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:04:12.051699 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:12.051706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:12.051763 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:12.051947 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:12.052830 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:12.054032 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:12.055337 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:12.055952 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:12.056536 (+ 584us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:12.059551 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:12.059557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:12.059557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:12.059572 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:12.061832 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:12.062369 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:12.062373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:12.062373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:12.062510 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:12.062769 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:12.062769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:12.062770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:12.063130 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:12.063739 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:12.063831 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 12:04:12.063941 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260619 12:04:12.065668 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:12.068873 22470 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:12.069571 22470 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:12.075407 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:12.067274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52971 (local address 127.0.203.254:36869)
0619 12:04:12.067312 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:12.067314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:12.067414 (+ 100us) server_negotiation.cc:407] Connection header received
0619 12:04:12.067468 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:12.067469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:12.067480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:12.067554 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:12.068231 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:12.069085 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:12.070076 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:12.070427 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:12.070531 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:12.071988 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:12.071990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:12.071990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:12.071997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:12.073770 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:12.074223 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:12.074224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:12.074225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:12.074287 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:12.074543 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:12.074544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:12.074545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:12.074674 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:12.074780 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:12.075217 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 12:04:12.075322 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 12:04:12.075517 22470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 006bd82992a241c59125d80646d21e1c: Generated new TSK 0
I20260619 12:04:12.075704 22470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:12.079373 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22388
W20260619 12:04:12.084522 22373 connection.cc:535] client connection to 127.0.203.254:36869 recv error: Network error: failed to read from TLS socket (remote: 127.0.203.254:36869): Connection reset by peer (error 104)
2026-06-19T12:04:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:12 dist-test-slave-fbq8 krb5kdc[22487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:12 dist-test-slave-fbq8 krb5kdc[22487](info): set up 2 sockets
Jun 19 12:04:12 dist-test-slave-fbq8 krb5kdc[22487](info): commencing operation
krb5kdc: starting...
W20260619 12:04:13.140938 815 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:13 dist-test-slave-fbq8 krb5kdc[22487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:13.280154 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:41481
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:13.369590 22500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:13.370227 22500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:13.370307 22500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:13.371769 22500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:13.372092 22500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:13.372124 22500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:13.372148 22500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:13.372179 22500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:13.374392 22500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:41481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:13.374907 22500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:13.375206 22500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:13.381220 22505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:13.381227 22506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:13.381220 22508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:13.381925 22500 server_base.cc:1034] running on GCE node
I20260619 12:04:13.382239 22500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:13.382580 22500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:13.383801 22500 hybrid_clock.cc:648] HybridClock initialized: now 1781870653383788 us; error 37 us; skew 500 ppm
Jun 19 12:04:13 dist-test-slave-fbq8 krb5kdc[22487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:13.386790 22500 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:13.387411 22500 webserver.cc:460] Webserver started at http://127.0.203.254:37369/ using document root <none> and password file <none>
I20260619 12:04:13.387667 22500 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:13.387745 22500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:13.387912 22500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:13.389137 22500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "47faa45059d946b3b01288da554cde91"
format_stamp: "Formatted at 2026-06-19 12:04:13 on dist-test-slave-fbq8"
server_key: "cf260da63cc8c1d480241c9560aec3fe"
server_key_iv: "0b56bf965b0bc272ea701e560dbbe5a7"
server_key_version: "encryptionkey@0"
I20260619 12:04:13.389518 22500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "47faa45059d946b3b01288da554cde91"
format_stamp: "Formatted at 2026-06-19 12:04:13 on dist-test-slave-fbq8"
server_key: "cf260da63cc8c1d480241c9560aec3fe"
server_key_iv: "0b56bf965b0bc272ea701e560dbbe5a7"
server_key_version: "encryptionkey@0"
I20260619 12:04:13.392604 22500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:04:13.393718 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:13.393991 22500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:13.394129 22500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "47faa45059d946b3b01288da554cde91"
format_stamp: "Formatted at 2026-06-19 12:04:13 on dist-test-slave-fbq8"
server_key: "cf260da63cc8c1d480241c9560aec3fe"
server_key_iv: "0b56bf965b0bc272ea701e560dbbe5a7"
server_key_version: "encryptionkey@0"
I20260619 12:04:13.394228 22500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:13.419132 22500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:13.421770 22500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:13.422027 22500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:13.427264 22500 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:41481
I20260619 12:04:13.427380 22567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:41481 every 8 connection(s)
I20260619 12:04:13.427809 22500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:13.429176 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:13.431910 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91: Bootstrap starting.
I20260619 12:04:13.432575 22568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:13.432936 22568 log.cc:824] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:13.433699 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91: No bootstrap required, opened a new log
I20260619 12:04:13.435403 22568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47faa45059d946b3b01288da554cde91" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41481 } }
I20260619 12:04:13.435563 22568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:13.435588 22568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47faa45059d946b3b01288da554cde91, State: Initialized, Role: FOLLOWER
I20260619 12:04:13.435748 22568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47faa45059d946b3b01288da554cde91" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41481 } }
I20260619 12:04:13.435837 22568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:13.435863 22568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:13.435894 22568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:13.436434 22568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47faa45059d946b3b01288da554cde91" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41481 } }
I20260619 12:04:13.436530 22568 leader_election.cc:302] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47faa45059d946b3b01288da554cde91; no voters:
I20260619 12:04:13.436779 22568 leader_election.cc:288] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:13.436810 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22500
I20260619 12:04:13.436936 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:13.437008 22571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:13.437197 815 external_mini_cluster.cc:1428] Setting key e50c278c16e2ebfeaa0e36bf4a84e9d4
I20260619 12:04:13.437209 22568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:13.437309 22571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [term 1 LEADER]: Becoming Leader. State: Replica: 47faa45059d946b3b01288da554cde91, State: Running, Role: LEADER
I20260619 12:04:13.437495 22571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47faa45059d946b3b01288da554cde91" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41481 } }
I20260619 12:04:13.438669 22573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47faa45059d946b3b01288da554cde91. Latest consensus state: current_term: 1 leader_uuid: "47faa45059d946b3b01288da554cde91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47faa45059d946b3b01288da554cde91" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41481 } } }
I20260619 12:04:13.438783 22573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:13.438977 22572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47faa45059d946b3b01288da554cde91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47faa45059d946b3b01288da554cde91" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41481 } } }
I20260619 12:04:13.439159 22583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:13.439193 22572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:13.439949 22583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:13.443565 22583 catalog_manager.cc:1347] Generated new cluster ID: 5936eac2834c4b2f8f0e3bb50f9e3a7d
I20260619 12:04:13.443632 22583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:13.450197 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:13.437807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60797 (local address 127.0.203.254:41481)
0619 12:04:13.439516 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:13.439528 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:13.439570 (+ 42us) server_negotiation.cc:407] Connection header received
0619 12:04:13.439824 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:13.439834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:13.439990 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:13.441212 (+ 1222us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:13.441334 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:13.443049 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:13.444229 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:13.444936 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:13.445067 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:13.449218 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:13.449223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:13.449223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:13.449475 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:13.449526 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:13.449544 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:13.449631 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:13.449804 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 12:04:13.449959 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":100,"threads_started":1}
I20260619 12:04:13.451470 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:13.455052 22583 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:13.455747 22583 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:13.457808 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:13.453275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45005 (local address 127.0.203.254:41481)
0619 12:04:13.453321 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:13.453322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:13.453591 (+ 269us) server_negotiation.cc:407] Connection header received
0619 12:04:13.453649 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:13.453650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:13.453668 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:13.453746 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:13.454364 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:13.455079 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:13.456094 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:13.456495 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:13.456603 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:13.457455 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:13.457456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:13.457456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:13.457462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:13.457483 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:13.457488 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:13.457563 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:13.457683 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 12:04:13.457701 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 12:04:13.460439 22583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47faa45059d946b3b01288da554cde91: Generated new TSK 0
I20260619 12:04:13.460759 22583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:13.466957 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22500
W20260619 12:04:13.471889 22485 connection.cc:535] client connection to 127.0.203.254:41481 recv error: Network error: failed to read from TLS socket (remote: 127.0.203.254:41481): Connection reset by peer (error 104)
2026-06-19T12:04:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:13 dist-test-slave-fbq8 krb5kdc[22599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:13 dist-test-slave-fbq8 krb5kdc[22599](info): set up 2 sockets
Jun 19 12:04:13 dist-test-slave-fbq8 krb5kdc[22599](info): commencing operation
krb5kdc: starting...
W20260619 12:04:14.537024 815 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:14 dist-test-slave-fbq8 krb5kdc[22599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:14.676811 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36951
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:36951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:14.761584 22612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:14.761854 22612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:14.761909 22612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:14.763186 22612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:14.763271 22612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:14.763296 22612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:14.763317 22612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:14.763356 22612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:14.765210 22612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:36951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:36951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:14.765522 22612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:14.765777 22612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:14.771260 22620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:14.771201 22618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:14.771240 22617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:14.771832 22612 server_base.cc:1034] running on GCE node
I20260619 12:04:14.772135 22612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:14.772469 22612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:14.773661 22612 hybrid_clock.cc:648] HybridClock initialized: now 1781870654773624 us; error 59 us; skew 500 ppm
Jun 19 12:04:14 dist-test-slave-fbq8 krb5kdc[22599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:14.776479 22612 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:14.777078 22612 webserver.cc:460] Webserver started at http://127.0.203.254:40609/ using document root <none> and password file <none>
I20260619 12:04:14.777318 22612 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:14.777385 22612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:14.777525 22612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:14.778620 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "45846453faff4c58ba3c1fefdf0280b3"
format_stamp: "Formatted at 2026-06-19 12:04:14 on dist-test-slave-fbq8"
server_key: "3bcb0a8416360b0816097d3334774542"
server_key_iv: "c72725816f6a987e772afd3e427a23ed"
server_key_version: "encryptionkey@0"
I20260619 12:04:14.778993 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "45846453faff4c58ba3c1fefdf0280b3"
format_stamp: "Formatted at 2026-06-19 12:04:14 on dist-test-slave-fbq8"
server_key: "3bcb0a8416360b0816097d3334774542"
server_key_iv: "c72725816f6a987e772afd3e427a23ed"
server_key_version: "encryptionkey@0"
I20260619 12:04:14.780455 22612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:04:14.781510 22627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:14.781734 22612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:14.781904 22612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "45846453faff4c58ba3c1fefdf0280b3"
format_stamp: "Formatted at 2026-06-19 12:04:14 on dist-test-slave-fbq8"
server_key: "3bcb0a8416360b0816097d3334774542"
server_key_iv: "c72725816f6a987e772afd3e427a23ed"
server_key_version: "encryptionkey@0"
I20260619 12:04:14.781986 22612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:14.785585 22612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:14.788000 22612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:14.788163 22612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:14.792800 22612 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:36951
I20260619 12:04:14.792954 22679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:36951 every 8 connection(s)
I20260619 12:04:14.793236 22612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:14.794572 22680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:14.797407 22680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3: Bootstrap starting.
I20260619 12:04:14.798048 22680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:14.798310 22680 log.cc:824] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:14.799249 22680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3: No bootstrap required, opened a new log
I20260619 12:04:14.801038 22680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45846453faff4c58ba3c1fefdf0280b3" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36951 } }
I20260619 12:04:14.801192 22680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:14.801218 22680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45846453faff4c58ba3c1fefdf0280b3, State: Initialized, Role: FOLLOWER
I20260619 12:04:14.801339 22680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45846453faff4c58ba3c1fefdf0280b3" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36951 } }
I20260619 12:04:14.801438 22680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:14.801476 22680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:14.801537 22680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:14.802156 22680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45846453faff4c58ba3c1fefdf0280b3" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36951 } }
I20260619 12:04:14.802290 22680 leader_election.cc:302] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45846453faff4c58ba3c1fefdf0280b3; no voters:
I20260619 12:04:14.802500 22680 leader_election.cc:288] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:14.802656 22683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:14.802856 22680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:14.802948 22683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [term 1 LEADER]: Becoming Leader. State: Replica: 45846453faff4c58ba3c1fefdf0280b3, State: Running, Role: LEADER
I20260619 12:04:14.802954 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22612
I20260619 12:04:14.803067 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:14.803144 22683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45846453faff4c58ba3c1fefdf0280b3" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36951 } }
I20260619 12:04:14.803261 815 external_mini_cluster.cc:1428] Setting key 11e120ae3c1c21223c2357191e5d6f68
I20260619 12:04:14.804141 22684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45846453faff4c58ba3c1fefdf0280b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45846453faff4c58ba3c1fefdf0280b3" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36951 } } }
I20260619 12:04:14.804270 22684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:14.804378 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45846453faff4c58ba3c1fefdf0280b3. Latest consensus state: current_term: 1 leader_uuid: "45846453faff4c58ba3c1fefdf0280b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45846453faff4c58ba3c1fefdf0280b3" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 36951 } } }
I20260619 12:04:14.804481 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3 [sys.catalog]: This master's current role is: LEADER
W20260619 12:04:14.805261 22701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:14.805356 22701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:14.805737 22699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:14.806011 22699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:14.806726 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:14.804096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59501 (local address 127.0.203.254:36951)
0619 12:04:14.804527 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:14.804539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:14.804946 (+ 407us) server_negotiation.cc:407] Connection header received
0619 12:04:14.805146 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:14.805156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:14.805193 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:14.805377 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:14.806258 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:14.806263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:14.806263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:14.806267 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:14.806315 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:14.806328 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:14.806415 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:14.806460 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 12:04:14.806505 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":110,"threads_started":1}
I20260619 12:04:14.807910 22699 catalog_manager.cc:1347] Generated new cluster ID: 9c3d2b5d3ba24b0d8b69af33845aa185
I20260619 12:04:14.807969 22699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:14.808024 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:14.814095 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:14.809778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53965 (local address 127.0.203.254:36951)
0619 12:04:14.809846 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:14.809848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:14.812880 (+ 3032us) server_negotiation.cc:407] Connection header received
0619 12:04:14.812926 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:14.812928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:14.812937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:14.813022 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:14.813723 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:14.813725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:14.813726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:14.813732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:14.813756 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:14.813762 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:14.813947 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:14.813982 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 12:04:14.814004 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:04:14.819841 22699 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:14.820098 22699 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:14.828953 22699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45846453faff4c58ba3c1fefdf0280b3: Generated new TSK 0
I20260619 12:04:14.829282 22699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:14.836601 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22612
2026-06-19T12:04:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:14 dist-test-slave-fbq8 krb5kdc[22711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:14 dist-test-slave-fbq8 krb5kdc[22711](info): set up 2 sockets
Jun 19 12:04:14 dist-test-slave-fbq8 krb5kdc[22711](info): commencing operation
krb5kdc: starting...
W20260619 12:04:15.905100 815 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:15 dist-test-slave-fbq8 krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:16.044480 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:41827
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:33735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:16.133289 22724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:16.133529 22724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:16.133553 22724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:16.134785 22724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:16.134853 22724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:16.134868 22724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:16.134882 22724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:16.134895 22724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:16.136466 22724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:33735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:41827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:16.136839 22724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:16.137141 22724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:16.142730 22729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:16.142737 22730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:16.142895 22724 server_base.cc:1034] running on GCE node
W20260619 12:04:16.142742 22732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:16.143361 22724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:16.143761 22724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:16.144968 22724 hybrid_clock.cc:648] HybridClock initialized: now 1781870656144941 us; error 52 us; skew 500 ppm
Jun 19 12:04:16 dist-test-slave-fbq8 krb5kdc[22711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:16.147997 22724 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:16.148523 22724 webserver.cc:460] Webserver started at http://127.0.203.254:40541/ using document root <none> and password file <none>
I20260619 12:04:16.148841 22724 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:16.148917 22724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:16.149085 22724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:16.150252 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "fedddee2584247efb59b313c5862958b"
format_stamp: "Formatted at 2026-06-19 12:04:16 on dist-test-slave-fbq8"
server_key: "38d652726b37bc77501ee971c4090614"
server_key_iv: "7f720b34c3d6afa52ae2c5ea3bed235c"
server_key_version: "encryptionkey@0"
I20260619 12:04:16.150671 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "fedddee2584247efb59b313c5862958b"
format_stamp: "Formatted at 2026-06-19 12:04:16 on dist-test-slave-fbq8"
server_key: "38d652726b37bc77501ee971c4090614"
server_key_iv: "7f720b34c3d6afa52ae2c5ea3bed235c"
server_key_version: "encryptionkey@0"
I20260619 12:04:16.152226 22724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:04:16.153303 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:16.153609 22724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 12:04:16.153743 22724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "fedddee2584247efb59b313c5862958b"
format_stamp: "Formatted at 2026-06-19 12:04:16 on dist-test-slave-fbq8"
server_key: "38d652726b37bc77501ee971c4090614"
server_key_iv: "7f720b34c3d6afa52ae2c5ea3bed235c"
server_key_version: "encryptionkey@0"
I20260619 12:04:16.153836 22724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:16.166559 22724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:16.169327 22724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:16.169513 22724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:16.174289 22724 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:41827
I20260619 12:04:16.174348 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:41827 every 8 connection(s)
I20260619 12:04:16.174865 22724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:16.176100 22792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:16.178994 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b: Bootstrap starting.
I20260619 12:04:16.179644 22792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:16.179926 22792 log.cc:824] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:16.181048 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b: No bootstrap required, opened a new log
I20260619 12:04:16.181056 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22724
I20260619 12:04:16.181178 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:16.181416 815 external_mini_cluster.cc:1428] Setting key 12fc7858411d965d7a34c35bee232c3e
I20260619 12:04:16.183411 22792 raft_consensus.cc:348] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedddee2584247efb59b313c5862958b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41827 } }
I20260619 12:04:16.183571 22792 raft_consensus.cc:374] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:16.183596 22792 raft_consensus.cc:729] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedddee2584247efb59b313c5862958b, State: Initialized, Role: FOLLOWER
I20260619 12:04:16.183740 22792 consensus_queue.cc:260] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedddee2584247efb59b313c5862958b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41827 } }
I20260619 12:04:16.183840 22792 raft_consensus.cc:388] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:16.183866 22792 raft_consensus.cc:482] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:16.183943 22792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:16.184970 22792 raft_consensus.cc:504] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedddee2584247efb59b313c5862958b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41827 } }
I20260619 12:04:16.185073 22792 leader_election.cc:302] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fedddee2584247efb59b313c5862958b; no voters:
I20260619 12:04:16.185343 22792 leader_election.cc:288] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:16.185369 22797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:16.185505 22797 raft_consensus.cc:686] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [term 1 LEADER]: Becoming Leader. State: Replica: fedddee2584247efb59b313c5862958b, State: Running, Role: LEADER
I20260619 12:04:16.185678 22792 sys_catalog.cc:564] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:16.185655 22797 consensus_queue.cc:237] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedddee2584247efb59b313c5862958b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41827 } }
I20260619 12:04:16.186076 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fedddee2584247efb59b313c5862958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedddee2584247efb59b313c5862958b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41827 } } }
I20260619 12:04:16.186156 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:16.186332 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fedddee2584247efb59b313c5862958b. Latest consensus state: current_term: 1 leader_uuid: "fedddee2584247efb59b313c5862958b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fedddee2584247efb59b313c5862958b" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 41827 } } }
I20260619 12:04:16.186412 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:16.186640 22803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:16.187320 22803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:16.189728 22803 catalog_manager.cc:1347] Generated new cluster ID: 853c6772cde2403398a06db5b434d784
I20260619 12:04:16.189816 22803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 12:04:16 dist-test-slave-fbq8 krb5kdc[22711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:16.195205 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:16.182296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52257 (local address 127.0.203.254:41827)
0619 12:04:16.182533 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:16.182551 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:16.182586 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:04:16.183432 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:16.183441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:16.183520 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:16.184541 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:16.184729 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:16.186024 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:16.187147 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:16.187883 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:16.188031 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:16.191088 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:16.191093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:16.191094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:16.191106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:16.193244 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:16.193642 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:16.193645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:16.193645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:16.193765 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:16.193939 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:16.193940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:16.193940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:16.194236 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:16.194384 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:16.194819 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 12:04:16.194967 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260619 12:04:16.196791 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:16.207067 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:16.198815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50865 (local address 127.0.203.254:41827)
0619 12:04:16.198854 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:16.198856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:16.198884 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:04:16.198971 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:16.198972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:16.198984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:16.199042 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:16.199831 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:16.200578 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:16.201732 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:16.202078 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:16.202147 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:16.203799 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:16.203801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:16.203801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:16.203809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:16.205758 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:16.206200 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:16.206201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:16.206202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:16.206264 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:16.206499 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:16.206500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:16.206501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:16.206628 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:16.206736 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:16.206884 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 12:04:16.206969 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 12:04:16.217713 22803 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:16.218307 22803 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:16.226838 22803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fedddee2584247efb59b313c5862958b: Generated new TSK 0
I20260619 12:04:16.227059 22803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:16.232583 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22724
2026-06-19T12:04:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:16 dist-test-slave-fbq8 krb5kdc[22823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:16 dist-test-slave-fbq8 krb5kdc[22823](info): set up 2 sockets
Jun 19 12:04:16 dist-test-slave-fbq8 krb5kdc[22823](info): commencing operation
krb5kdc: starting...
W20260619 12:04:17.284966 815 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:17 dist-test-slave-fbq8 krb5kdc[22823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:17.420734 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33897
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:17.508610 22836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:17.508955 22836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:17.508999 22836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:17.510217 22836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:17.510290 22836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:17.510324 22836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:17.510353 22836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:17.510380 22836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:17.511897 22836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:17.512190 22836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:17.512439 22836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:17.518381 22841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:17.518602 22842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:17.518371 22844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:17.518769 22836 server_base.cc:1034] running on GCE node
I20260619 12:04:17.519042 22836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:17.519377 22836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:17.520582 22836 hybrid_clock.cc:648] HybridClock initialized: now 1781870657520574 us; error 55 us; skew 500 ppm
Jun 19 12:04:17 dist-test-slave-fbq8 krb5kdc[22823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:17.528426 22836 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:17.529076 22836 webserver.cc:460] Webserver started at http://127.0.203.254:32941/ using document root <none> and password file <none>
I20260619 12:04:17.529332 22836 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:17.529409 22836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:17.529572 22836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:17.530681 22836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "11477079dddd4b4c9bacb4c08260e9e1"
format_stamp: "Formatted at 2026-06-19 12:04:17 on dist-test-slave-fbq8"
server_key: "c3cbe2ae398e62727c1f6a04878bd222"
server_key_iv: "48637f7a0d1a5d5ac385c75d0acf68f6"
server_key_version: "encryptionkey@0"
I20260619 12:04:17.531062 22836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "11477079dddd4b4c9bacb4c08260e9e1"
format_stamp: "Formatted at 2026-06-19 12:04:17 on dist-test-slave-fbq8"
server_key: "c3cbe2ae398e62727c1f6a04878bd222"
server_key_iv: "48637f7a0d1a5d5ac385c75d0acf68f6"
server_key_version: "encryptionkey@0"
I20260619 12:04:17.532539 22836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:04:17.533629 22851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:17.533946 22836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:17.534071 22836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "11477079dddd4b4c9bacb4c08260e9e1"
format_stamp: "Formatted at 2026-06-19 12:04:17 on dist-test-slave-fbq8"
server_key: "c3cbe2ae398e62727c1f6a04878bd222"
server_key_iv: "48637f7a0d1a5d5ac385c75d0acf68f6"
server_key_version: "encryptionkey@0"
I20260619 12:04:17.534159 22836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:17.553524 22836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:17.556047 22836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:17.556231 22836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:17.561355 22836 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:33897
I20260619 12:04:17.561511 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:33897 every 8 connection(s)
I20260619 12:04:17.561841 22836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:17.562915 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:17.565732 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1: Bootstrap starting.
I20260619 12:04:17.566425 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:17.566766 22904 log.cc:824] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:17.567166 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22836
I20260619 12:04:17.567312 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:17.567517 815 external_mini_cluster.cc:1428] Setting key e9e1c88413a448585635402eada1f808
I20260619 12:04:17.567859 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1: No bootstrap required, opened a new log
I20260619 12:04:17.569916 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11477079dddd4b4c9bacb4c08260e9e1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33897 } }
I20260619 12:04:17.570101 22904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:17.570142 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11477079dddd4b4c9bacb4c08260e9e1, State: Initialized, Role: FOLLOWER
I20260619 12:04:17.570334 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11477079dddd4b4c9bacb4c08260e9e1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33897 } }
I20260619 12:04:17.570437 22904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:17.570499 22904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:17.570560 22904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:17.571178 22904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11477079dddd4b4c9bacb4c08260e9e1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33897 } }
I20260619 12:04:17.571308 22904 leader_election.cc:302] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11477079dddd4b4c9bacb4c08260e9e1; no voters:
I20260619 12:04:17.571516 22904 leader_election.cc:288] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:17.571593 22909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:17.571837 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:17.571950 22909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 11477079dddd4b4c9bacb4c08260e9e1, State: Running, Role: LEADER
I20260619 12:04:17.572094 22909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11477079dddd4b4c9bacb4c08260e9e1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33897 } }
I20260619 12:04:17.574147 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11477079dddd4b4c9bacb4c08260e9e1. Latest consensus state: current_term: 1 leader_uuid: "11477079dddd4b4c9bacb4c08260e9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11477079dddd4b4c9bacb4c08260e9e1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33897 } } }
I20260619 12:04:17.574276 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [sys.catalog]: This master's current role is: LEADER
W20260619 12:04:17.574283 22924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 12:04:17.574138 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11477079dddd4b4c9bacb4c08260e9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11477079dddd4b4c9bacb4c08260e9e1" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 33897 } } }
W20260619 12:04:17.574409 22924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:17.574421 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:17.576726 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:17.568230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53439 (local address 127.0.203.254:33897)
0619 12:04:17.568476 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:17.568486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:17.568526 (+ 40us) server_negotiation.cc:407] Connection header received
0619 12:04:17.569932 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:17.569943 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:17.570021 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:17.570221 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:17.570481 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:17.572218 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:17.573781 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:17.574727 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:17.574874 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:17.575818 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:17.575823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:17.575823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:17.576069 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:17.576117 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:17.576134 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:17.576222 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:17.576327 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 12:04:17.576405 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260619 12:04:17.576857 22926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:17.577389 22926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:17.577924 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:17.579686 22926 catalog_manager.cc:1347] Generated new cluster ID: 6e59f7c48e4f4d6a9d57500d55f13164
I20260619 12:04:17.579756 22926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:17.583124 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:17.579917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49495 (local address 127.0.203.254:33897)
0619 12:04:17.580062 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:17.580064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:17.580127 (+ 63us) server_negotiation.cc:407] Connection header received
0619 12:04:17.580232 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:17.580233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:17.580245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:17.580297 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:17.580458 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:17.581037 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:17.581774 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:17.581973 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:17.582058 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:17.582810 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:17.582812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:17.582813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:17.582818 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:17.582834 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:17.582839 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:17.582911 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:17.582972 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 12:04:17.583001 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260619 12:04:17.588531 22926 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:17.589251 22926 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:17.595323 22926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11477079dddd4b4c9bacb4c08260e9e1: Generated new TSK 0
I20260619 12:04:17.595602 22926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:17.604010 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22836
2026-06-19T12:04:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:17 dist-test-slave-fbq8 krb5kdc[22935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:17 dist-test-slave-fbq8 krb5kdc[22935](info): set up 2 sockets
Jun 19 12:04:17 dist-test-slave-fbq8 krb5kdc[22935](info): commencing operation
krb5kdc: starting...
W20260619 12:04:18.677172 815 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:18 dist-test-slave-fbq8 krb5kdc[22935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:18.813753 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40243
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:39153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:40243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:18.922518 22948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:18.922811 22948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:18.922856 22948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:18.924292 22948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:18.924367 22948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:18.924391 22948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:18.924420 22948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:18.924451 22948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:18.926119 22948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:39153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:40243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:18.926427 22948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:18.926671 22948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:18.932158 22953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:18.932140 22954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:18.932411 22948 server_base.cc:1034] running on GCE node
W20260619 12:04:18.932147 22956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:18.932811 22948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:18.933204 22948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:18.934393 22948 hybrid_clock.cc:648] HybridClock initialized: now 1781870658934371 us; error 46 us; skew 500 ppm
Jun 19 12:04:18 dist-test-slave-fbq8 krb5kdc[22935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:18.937487 22948 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:18.938097 22948 webserver.cc:460] Webserver started at http://127.0.203.254:37203/ using document root <none> and password file <none>
I20260619 12:04:18.938361 22948 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:18.938436 22948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:18.938591 22948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:18.939729 22948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "04ea47541e4847caad2f83097ccaf0fa"
format_stamp: "Formatted at 2026-06-19 12:04:18 on dist-test-slave-fbq8"
server_key: "9bef27e5316813d0d134df33343efd11"
server_key_iv: "89076d1dbca482314a812083a39323f7"
server_key_version: "encryptionkey@0"
I20260619 12:04:18.940116 22948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "04ea47541e4847caad2f83097ccaf0fa"
format_stamp: "Formatted at 2026-06-19 12:04:18 on dist-test-slave-fbq8"
server_key: "9bef27e5316813d0d134df33343efd11"
server_key_iv: "89076d1dbca482314a812083a39323f7"
server_key_version: "encryptionkey@0"
I20260619 12:04:18.941745 22948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:04:18.942981 22963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:18.943487 22948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:04:18.943629 22948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "04ea47541e4847caad2f83097ccaf0fa"
format_stamp: "Formatted at 2026-06-19 12:04:18 on dist-test-slave-fbq8"
server_key: "9bef27e5316813d0d134df33343efd11"
server_key_iv: "89076d1dbca482314a812083a39323f7"
server_key_version: "encryptionkey@0"
I20260619 12:04:18.943722 22948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:18.947896 22948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:18.950307 22948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:18.950524 22948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:18.955183 22948 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:40243
I20260619 12:04:18.955240 23015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:40243 every 8 connection(s)
I20260619 12:04:18.955622 22948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:18.957075 23016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:18.959699 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 22948
I20260619 12:04:18.959807 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:18.960039 815 external_mini_cluster.cc:1428] Setting key b1c50dcf1b4239fafb1ef5191e14d73b
I20260619 12:04:18.960188 23016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa: Bootstrap starting.
I20260619 12:04:18.960999 23016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:18.961483 23016 log.cc:824] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:18.962522 23016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa: No bootstrap required, opened a new log
I20260619 12:04:18.963333 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:18.960720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37285 (local address 127.0.203.254:40243)
0619 12:04:18.960973 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:18.960983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:18.961015 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:04:18.961735 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:18.961744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:18.961774 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:18.961968 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:18.962854 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:18.962857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:18.962858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:18.962862 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:18.962915 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:18.962926 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:18.963030 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:18.963091 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 12:04:18.963137 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260619 12:04:18.964874 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:18.965384 23016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ea47541e4847caad2f83097ccaf0fa" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40243 } }
I20260619 12:04:18.965564 23016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:04:18.965611 23016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ea47541e4847caad2f83097ccaf0fa, State: Initialized, Role: FOLLOWER
I20260619 12:04:18.965741 23016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ea47541e4847caad2f83097ccaf0fa" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40243 } }
I20260619 12:04:18.965855 23016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:04:18.965902 23016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:04:18.965953 23016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:18.966617 23016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ea47541e4847caad2f83097ccaf0fa" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40243 } }
I20260619 12:04:18.966737 23016 leader_election.cc:302] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ea47541e4847caad2f83097ccaf0fa; no voters:
I20260619 12:04:18.966917 23016 leader_election.cc:288] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:04:18.966982 23026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:18.967231 23016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:18.967272 23026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [term 1 LEADER]: Becoming Leader. State: Replica: 04ea47541e4847caad2f83097ccaf0fa, State: Running, Role: LEADER
I20260619 12:04:18.967936 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:18.966594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54807 (local address 127.0.203.254:40243)
0619 12:04:18.966663 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:18.966667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:18.966699 (+ 32us) server_negotiation.cc:407] Connection header received
0619 12:04:18.966783 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:18.966784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:18.966790 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:18.966835 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:18.967653 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:18.967655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:18.967655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:04:18.967660 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:18.967674 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:04:18.967678 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:18.967725 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:18.967827 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 12:04:18.967842 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:04:18.967545 23026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ea47541e4847caad2f83097ccaf0fa" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40243 } }
I20260619 12:04:18.968818 23027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04ea47541e4847caad2f83097ccaf0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ea47541e4847caad2f83097ccaf0fa" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40243 } } }
I20260619 12:04:18.968930 23027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:18.969069 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ea47541e4847caad2f83097ccaf0fa. Latest consensus state: current_term: 1 leader_uuid: "04ea47541e4847caad2f83097ccaf0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ea47541e4847caad2f83097ccaf0fa" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40243 } } }
I20260619 12:04:18.969377 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa [sys.catalog]: This master's current role is: LEADER
W20260619 12:04:18.969434 23040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:18.969507 23040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:18.969597 23043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:18.969837 23043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:18.971714 23043 catalog_manager.cc:1347] Generated new cluster ID: 872741a9b88d4465ad2eacaada785d66
I20260619 12:04:18.971794 23043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:18.992523 23043 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:18.992843 23043 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:18.998028 23043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04ea47541e4847caad2f83097ccaf0fa: Generated new TSK 0
I20260619 12:04:18.998337 23043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:19.011674 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 22948
2026-06-19T12:04:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1410 ms)
[----------] 18 tests from AuthTokenIssuingTest (25042 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:19 dist-test-slave-fbq8 krb5kdc[23047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:19 dist-test-slave-fbq8 krb5kdc[23047](info): set up 2 sockets
Jun 19 12:04:19 dist-test-slave-fbq8 krb5kdc[23047](info): commencing operation
krb5kdc: starting...
W20260619 12:04:20.105206 815 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:20.252982 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40019
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:40019,127.0.203.253:42131,127.0.203.252:41661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:20.339733 23060 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:04:20.340008 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:20.340037 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:20.340080 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:20.341320 23060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:20.341398 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:20.341425 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:20.341446 23060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:20.341477 23060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:20.343029 23060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:40019,127.0.203.253:42131,127.0.203.252:41661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:40019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:20.343323 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:20.343569 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:20.349119 23065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:20.349298 23060 server_base.cc:1034] running on GCE node
W20260619 12:04:20.349119 23068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:20.349148 23066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:20.349725 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:20.350076 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:20.351274 23060 hybrid_clock.cc:648] HybridClock initialized: now 1781870660351262 us; error 56 us; skew 500 ppm
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:20.354483 23060 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:20.354965 23060 webserver.cc:460] Webserver started at http://127.0.203.254:42909/ using document root <none> and password file <none>
I20260619 12:04:20.355238 23060 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:20.355286 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:20.355423 23060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:20.356905 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "f02c34e9be964900a8cdd86e7d0461d6"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "3778156b45a33edff15641241acf0682"
server_key_iv: "34e61cf2e5c6407e6aa07f98a4b96f2f"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.357375 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "f02c34e9be964900a8cdd86e7d0461d6"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "3778156b45a33edff15641241acf0682"
server_key_iv: "34e61cf2e5c6407e6aa07f98a4b96f2f"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.359079 23060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 12:04:20.360177 23075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:20.360453 23060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:04:20.360572 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "f02c34e9be964900a8cdd86e7d0461d6"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "3778156b45a33edff15641241acf0682"
server_key_iv: "34e61cf2e5c6407e6aa07f98a4b96f2f"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.360729 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:20.381285 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:20.384095 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:20.384339 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:20.389307 23060 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:40019
I20260619 12:04:20.389375 23127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:40019 every 8 connection(s)
I20260619 12:04:20.389843 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:20.391094 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:20.393076 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:20.399794 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 23060
I20260619 12:04:20.399936 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:20.400159 815 external_mini_cluster.cc:1428] Setting key 1d523f416f8914f5db7c6b0e30e52ca8
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:20.405236 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.393955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40019 (local address 127.0.0.1:46666)
0619 12:04:20.394171 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.394340 (+ 169us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.394935 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.395282 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.395284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.395602 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.395776 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.395780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.396978 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.396979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.397667 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.397669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.398215 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.398216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.398281 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.398903 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.398913 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.400952 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.403119 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.403129 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.403132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.403463 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.403833 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.403834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.403834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.403933 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.404436 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.404442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.404523 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.404615 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.405001 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":130,"thread_start_us":95,"threads_started":1}
I20260619 12:04:20.405246 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.394158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.0.203.254:40019)
0619 12:04:20.394396 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.394402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.394482 (+ 80us) server_negotiation.cc:407] Connection header received
0619 12:04:20.395131 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.395134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.395161 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.395263 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.395885 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.396864 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.397770 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.398087 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.398220 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.401070 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.401079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.401079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.401087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.402997 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.403566 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.403569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.403570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.403684 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.404011 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.404012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.404013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.404345 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.404437 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.404702 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.404869 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":48,"threads_started":1}
I20260619 12:04:20.406327 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:20.406864 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.406643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55050)
0619 12:04:20.406711 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.406776 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 12:04:20.407084 23079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
I20260619 12:04:20.407618 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.407495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55060)
0619 12:04:20.407556 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.407593 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:04:20.407797 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:42131: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.203.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab.
Entry for principal kudu/127.0.203.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab.
I20260619 12:04:20.428507 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:42131
--webserver_interface=127.0.203.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab
--principal=kudu/127.0.203.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:40019,127.0.203.253:42131,127.0.203.252:41661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:04:20.457093 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } attempt: 1
I20260619 12:04:20.457677 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.457534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55076)
0619 12:04:20.457607 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.457650 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 12:04:20.458025 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.457932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55090)
0619 12:04:20.457977 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.458001 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 12:04:20.458109 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:42131: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
I20260619 12:04:20.526288 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } attempt: 2
I20260619 12:04:20.527012 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.526784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55106)
0619 12:04:20.526920 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.526981 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 12:04:20.527534 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.527373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55114)
0619 12:04:20.527476 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.527513 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 12:04:20.527717 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:42131: Network error: Client connection negotiation failed: client connection to 127.0.203.253:42131: connect: Connection refused (error 111)
W20260619 12:04:20.529819 23135 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:04:20.530078 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:20.530121 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:20.530174 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:20.531418 23135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:20.531482 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:20.531508 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:20.531538 23135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:20.531564 23135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:20.533212 23135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:40019,127.0.203.253:42131,127.0.203.252:41661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab
--principal=kudu/127.0.203.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:42131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:20.533488 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:20.533687 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:20.539448 23143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:20.539474 23141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:20.539630 23135 server_base.cc:1034] running on GCE node
W20260619 12:04:20.539474 23140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:20.540155 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:20.540717 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:20.541999 23135 hybrid_clock.cc:648] HybridClock initialized: now 1781870660541965 us; error 54 us; skew 500 ppm
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:20.545754 23135 init.cc:377] Logged in from keytab as kudu/127.0.203.253@KRBTEST.COM (short username kudu)
I20260619 12:04:20.546309 23135 webserver.cc:460] Webserver started at http://127.0.203.253:35137/ using document root <none> and password file <none>
I20260619 12:04:20.546586 23135 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:20.546640 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:20.546772 23135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:20.547928 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data/instance:
uuid: "2c7377bd926446afa987f08b36956045"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "e25123c6bab1ee55c00a005e18ebb4b5"
server_key_iv: "b2e354c5015c93fbf13ef07fcd47c021"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.548295 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/wal/instance:
uuid: "2c7377bd926446afa987f08b36956045"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "e25123c6bab1ee55c00a005e18ebb4b5"
server_key_iv: "b2e354c5015c93fbf13ef07fcd47c021"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.549816 23135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 12:04:20.550800 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:20.551040 23135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:04:20.551160 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/wal
uuid: "2c7377bd926446afa987f08b36956045"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "e25123c6bab1ee55c00a005e18ebb4b5"
server_key_iv: "b2e354c5015c93fbf13ef07fcd47c021"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.551240 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:20.565909 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:20.568426 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:20.568624 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:20.573299 23135 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.253:42131
I20260619 12:04:20.573422 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.253:42131 every 8 connection(s)
I20260619 12:04:20.573769 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
I20260619 12:04:20.574987 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 23135
I20260619 12:04:20.575142 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-1/wal/instance
I20260619 12:04:20.575137 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:20.575368 815 external_mini_cluster.cc:1428] Setting key c87b09ec909bc47fea202a7432c19e9f
I20260619 12:04:20.577924 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.252@KRBTEST.COM" created.
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:20.592001 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.578823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.0.203.254:40019)
0619 12:04:20.578892 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.578894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.579187 (+ 293us) server_negotiation.cc:407] Connection header received
0619 12:04:20.579864 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.579867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.579889 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.579986 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.580866 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.581721 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.582901 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.583267 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.583362 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.588290 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.588292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.588292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.588308 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.590163 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.590741 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.590742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.590742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.590806 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.591113 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.591114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.591114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.591242 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.591359 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.591806 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.591913 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:04:20.592206 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.578794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40019 (local address 127.0.0.1:46672)
0619 12:04:20.579030 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.579076 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.579668 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.580128 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.580132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.580424 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.580754 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.580760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.581864 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.581865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.582791 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.582795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.583390 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.583391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.583530 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.584234 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.584243 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.588159 (+ 3916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.590271 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.590281 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.590285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.590657 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.590877 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.590878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.590878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.591028 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.591363 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.591371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.591575 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.591711 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.591933 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":147,"threads_started":1}
I20260619 12:04:20.593042 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:04:20.604576 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.593434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55126)
0619 12:04:20.593503 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.593514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.593557 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.593983 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.593984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.594149 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.594227 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.594229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.595213 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.595215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.596112 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.596114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.596828 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.596829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.596906 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.597529 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.597534 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.599988 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.602602 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.602604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.602605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.602964 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.603260 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.603261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.603261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.603369 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.603946 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.603948 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.604042 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.604161 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.604323 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 12:04:20.604610 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.593512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.203.253:42131)
0619 12:04:20.593781 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.593786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.593813 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:04:20.593837 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.593839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.593853 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.593944 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.594359 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.595104 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.596230 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.596728 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.596818 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.600092 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.600102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.600102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.600111 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.602457 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.603093 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.603098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.603098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.603176 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.603441 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.603442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.603443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.603809 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.603952 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.604234 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.604395 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":123,"threads_started":1}
I20260619 12:04:20.605692 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:20.606177 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.606022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52252)
0619 12:04:20.606087 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.606144 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 12:04:20.606405 23151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab.
Entry for principal kudu/127.0.203.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab.
I20260619 12:04:20.607038 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.606844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52254)
0619 12:04:20.606981 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.607012 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260619 12:04:20.607215 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41661: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
I20260619 12:04:20.607880 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:41661
--webserver_interface=127.0.203.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab
--principal=kudu/127.0.203.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:40019,127.0.203.253:42131,127.0.203.252:41661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:04:20.618949 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } attempt: 3
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:04:20.627255 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.619511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55138)
0619 12:04:20.619609 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.619622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.619679 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.619951 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.619954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.620121 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.620203 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.620205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.621344 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.621345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.621898 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.621900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.622412 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.622412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.622470 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.622979 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.622985 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.624261 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.626185 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.626186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.626187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.626371 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.626599 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.626600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.626601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.626655 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.626982 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.626982 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.627038 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.627099 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.627182 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 12:04:20.627348 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.619569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.0.203.253:42131)
0619 12:04:20.619777 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.619780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.619790 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:04:20.619823 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.619825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.619841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.619906 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.620336 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.621247 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.621983 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.622331 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.622410 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.624352 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.624353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.624353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.624361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.626091 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.626453 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.626454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.626455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.626519 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.626732 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.626733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.626733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.626903 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.626990 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.627152 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.627238 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260619 12:04:20.627841 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:20.628268 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.628140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52258)
0619 12:04:20.628204 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.628242 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 12:04:20.628741 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.628623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52266)
0619 12:04:20.628692 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.628719 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 12:04:20.628868 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41661: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
I20260619 12:04:20.656492 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } attempt: 1
I20260619 12:04:20.657198 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.657004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52280)
0619 12:04:20.657093 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.657158 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 12:04:20.657760 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.657645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52284)
0619 12:04:20.657710 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.657741 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 12:04:20.658034 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41661: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
I20260619 12:04:20.660040 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } attempt: 1
I20260619 12:04:20.660749 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.660497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52296)
0619 12:04:20.660592 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.660718 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 12:04:20.661250 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.661123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52298)
0619 12:04:20.661197 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.661231 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 12:04:20.661422 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41661: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
W20260619 12:04:20.706755 23210 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:04:20.707041 23210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:20.707069 23210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:20.707113 23210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:20.708251 23210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:20.708317 23210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:20.708343 23210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:20.708377 23210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:20.708408 23210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:20.709995 23210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:40019,127.0.203.253:42131,127.0.203.252:41661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab
--principal=kudu/127.0.203.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:41661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:20.710317 23210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:20.710561 23210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:20.715986 23215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:20.715982 23218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:20.715983 23216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:20.716373 23210 server_base.cc:1034] running on GCE node
I20260619 12:04:20.716722 23210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:20.717098 23210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:20.718271 23210 hybrid_clock.cc:648] HybridClock initialized: now 1781870660718257 us; error 46 us; skew 500 ppm
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:20.721210 23210 init.cc:377] Logged in from keytab as kudu/127.0.203.252@KRBTEST.COM (short username kudu)
I20260619 12:04:20.721772 23210 webserver.cc:460] Webserver started at http://127.0.203.252:44071/ using document root <none> and password file <none>
I20260619 12:04:20.722033 23210 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:20.722114 23210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:20.722272 23210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:20.723412 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data/instance:
uuid: "4756a01cd940444581e7faf53e95cadf"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "bc93c352ff398c36bec997f68c31aa6e"
server_key_iv: "2a056c10104a2b0c813a1a83933c8e8d"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.723801 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/wal/instance:
uuid: "4756a01cd940444581e7faf53e95cadf"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "bc93c352ff398c36bec997f68c31aa6e"
server_key_iv: "2a056c10104a2b0c813a1a83933c8e8d"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.725381 23210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 12:04:20.726213 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } attempt: 2
I20260619 12:04:20.726512 23225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:20.726949 23210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:20.726974 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.726719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52300)
0619 12:04:20.726858 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.726922 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 12:04:20.727067 23210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/wal
uuid: "4756a01cd940444581e7faf53e95cadf"
format_stamp: "Formatted at 2026-06-19 12:04:20 on dist-test-slave-fbq8"
server_key: "bc93c352ff398c36bec997f68c31aa6e"
server_key_iv: "2a056c10104a2b0c813a1a83933c8e8d"
server_key_version: "encryptionkey@0"
I20260619 12:04:20.727188 23210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:20.727499 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.727350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52304)
0619 12:04:20.727459 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.727482 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 12:04:20.727602 23203 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41661: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
I20260619 12:04:20.736613 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } attempt: 2
I20260619 12:04:20.737167 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.737024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52320)
0619 12:04:20.737090 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.737133 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 12:04:20.737555 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.737454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52328)
0619 12:04:20.737501 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.737527 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 12:04:20.737644 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41661: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41661: connect: Connection refused (error 111)
I20260619 12:04:20.740912 23210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:20.743006 23210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:20.743189 23210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:20.748574 23210 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.252:41661
I20260619 12:04:20.748629 23277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.252:41661 every 8 connection(s)
I20260619 12:04:20.749053 23210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
I20260619 12:04:20.750228 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:20.751938 23278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:20.754992 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 23210
I20260619 12:04:20.755131 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781870380032316-815-0/minicluster-data/master-2/wal/instance
I20260619 12:04:20.755342 815 external_mini_cluster.cc:1428] Setting key 96b9e978d513a61c94e3bddca61b8044
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:04:20.763271 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.752799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.0.203.254:40019)
0619 12:04:20.752874 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.752876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.753083 (+ 207us) server_negotiation.cc:407] Connection header received
0619 12:04:20.753760 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.753761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.753775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.753868 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.754597 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.755317 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.756959 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.757164 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.757278 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.759983 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.759984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.759985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.759993 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.761572 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.762113 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.762114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.762115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.762183 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.762451 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.762452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.762452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.762591 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.762648 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.763053 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.763167 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:04:20.763482 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.752742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:40019 (local address 127.0.0.1:46682)
0619 12:04:20.752916 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.752968 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.753619 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.753953 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.753956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.754247 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.754472 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.754476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.755474 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.755475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.756853 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.756857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.757327 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.757327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.757439 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.758051 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.758060 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.759838 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.761674 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.761682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.761684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.762032 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.762261 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.762261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.762261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.762374 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.762675 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.762681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.762857 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.763004 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.763193 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260619 12:04:20.764328 23278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:20.765775 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.755938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.203.252:41661)
0619 12:04:20.756161 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.756167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.756283 (+ 116us) server_negotiation.cc:407] Connection header received
0619 12:04:20.756317 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.756319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.756344 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.756432 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.757251 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.758248 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.759164 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.759549 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.759638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.762479 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.762487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.762487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.762494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.764481 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.764940 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.764945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.764946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.765013 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.765232 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.765233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.765234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.765407 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.765520 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.765636 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.765700 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":143,"threads_started":1}
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:20.772246 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.764647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:42131 (local address 127.0.0.1:55150)
0619 12:04:20.764720 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.764734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.764776 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.764953 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.764954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.765116 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.765173 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.765175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.766038 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.766039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.766709 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.766712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.767127 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.767128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.767185 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.767617 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.767622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.768941 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.770963 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.770964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.770965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.771156 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.771374 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.771375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.771376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.771436 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.771818 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.771819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.771868 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.772010 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.772120 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 12:04:20.772238 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.764742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.0.203.253:42131)
0619 12:04:20.764816 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.764819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.764830 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:04:20.764860 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.764861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.764876 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.764936 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.765282 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.765967 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.766796 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.767033 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.767090 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.769040 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.769041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.769041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.769056 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.770835 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.771244 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.771245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.771245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.771309 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.771563 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.771564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.771565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.771724 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.771810 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.772076 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.772159 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:04:20.772994 23278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:20.775199 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.766759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.0.203.254:40019)
0619 12:04:20.767146 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.767147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.767153 (+ 6us) server_negotiation.cc:407] Connection header received
0619 12:04:20.767170 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.767171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.767179 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.767245 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.768069 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.768525 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.769380 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.769582 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.769668 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.772010 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.772013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.772013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.772023 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.773825 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.774236 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.774237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.774237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.774307 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.774561 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.774562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.774563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.774711 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.774967 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.775009 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.775095 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367}
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:04:20.780548 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.773337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52344)
0619 12:04:20.773377 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.773388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.773429 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.773557 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.773558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.773712 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.773748 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.773750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.774634 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.774635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.775136 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.775137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.775649 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.775650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.775779 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.776142 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.776146 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.777664 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.779420 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.779421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.779422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.779601 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.779814 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.779815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.779815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.779875 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.780114 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.780115 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.780156 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.780221 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.780360 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":2}
I20260619 12:04:20.780566 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.773393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.0.203.252:41661)
0619 12:04:20.773440 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.773442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.773451 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:04:20.773472 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.773473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.773485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.773556 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.773833 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.774555 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.775223 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.775566 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.775642 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.777778 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.777780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.777780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.777787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.779335 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.779684 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.779685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.779686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.779740 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.779937 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.779937 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.779938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.780047 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.780140 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.780240 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.780446 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":27}
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:04:20.782240 23278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: Bootstrap starting.
I20260619 12:04:20.783115 23278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:20.783416 23278 log.cc:824] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:20.784178 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.776167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.0.203.253:42131)
0619 12:04:20.776225 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.776227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.776235 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:04:20.776254 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.776255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.776265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.776321 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.776995 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.777674 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.778609 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.778960 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.779044 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.781515 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.781516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.781517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.781524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.782988 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.783427 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.783428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.783429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.783484 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.783687 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.783687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.783688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.783798 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.783965 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.784002 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.784078 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 12:04:20.784196 23278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: No bootstrap required, opened a new log
I20260619 12:04:20.786132 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:20.786375 23278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.786520 23278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4756a01cd940444581e7faf53e95cadf, State: Initialized, Role: FOLLOWER
I20260619 12:04:20.786615 23278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.786938 23278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:20.786958 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.787088 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:20.787686 23102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f02c34e9be964900a8cdd86e7d0461d6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684
W20260619 12:04:20.788961 23295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:20.789057 23295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:20.789439 23210 master_runner.cc:186] Error getting master registration for 127.0.203.254:40019: OK, instance_id { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" instance_seqno: 1781870660387903 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:04:20.792671 23210 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:04:20.818759 23203 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } attempt: 3
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:04:20.829128 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.819329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52358)
0619 12:04:20.819401 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.819417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.819475 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.819812 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.819813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.820036 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.820108 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.820109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.821227 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.821228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.822101 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.822102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.822631 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.822631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.822696 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.823338 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.823343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.825422 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.827550 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.827551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.827552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.827813 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.828053 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.828054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.828055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.828123 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.828765 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.828766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.828825 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.828898 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.829012 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 12:04:20.829193 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.819488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.0.203.252:41661)
0619 12:04:20.819587 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.819590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.819604 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:04:20.819644 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.819646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.819660 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.819786 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.820235 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.821110 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.822212 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.822538 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.822622 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.825552 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.825554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.825554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.825562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.827445 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.827904 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.827905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.827906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.827972 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.828199 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.828200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.828201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.828584 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.828774 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.828961 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.829082 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:04:20.831830 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: Bootstrap starting.
I20260619 12:04:20.832808 23203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:20.833154 23203 log.cc:824] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:20.834270 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: No bootstrap required, opened a new log
I20260619 12:04:20.836596 23203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.836815 23203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7377bd926446afa987f08b36956045, State: Initialized, Role: FOLLOWER
I20260619 12:04:20.836781 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } attempt: 3
I20260619 12:04:20.836921 23203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.837322 23203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:20.837750 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.837899 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 12:04:20.841152 23309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:20.841218 23309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 12:04:20 dist-test-slave-fbq8 krb5kdc[23047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:04:20.839810 23135 master_runner.cc:186] Error getting master registration for 127.0.203.254:40019: OK, instance_id { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" instance_seqno: 1781870660387903 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:04:20.843402 23135 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:04:20.845028 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.837392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.0.203.252:41661)
0619 12:04:20.837447 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.837449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.837456 (+ 7us) server_negotiation.cc:407] Connection header received
0619 12:04:20.837484 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.837484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.837495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.837548 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.837911 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.838545 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.839539 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.839824 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.839881 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.842057 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.842059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.842059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.842067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.843749 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.844144 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.844145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.844145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.844232 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.844423 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.844424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.844424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.844579 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.844662 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.844845 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.844929 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 12:04:20.845028 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.837173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41661 (local address 127.0.0.1:52374)
0619 12:04:20.837252 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:20.837266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:20.837315 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:20.837552 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:20.837553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:20.837763 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:20.837818 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.837820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.838632 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.838633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.839441 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:20.839443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.839878 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.839879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:20.839955 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.840478 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:20.840484 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:20.841974 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:20.843822 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.843824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.843824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.844060 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.844288 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:20.844288 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:20.844289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:20.844363 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:20.844689 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:20.844690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:20.844740 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 12:04:20.844807 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:04:20.844918 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 12:04:20.846711 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6: Bootstrap starting.
I20260619 12:04:20.847529 23128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:20.847824 23128 log.cc:824] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:20.848663 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6: No bootstrap required, opened a new log
I20260619 12:04:20.850163 23128 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.850311 23128 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02c34e9be964900a8cdd86e7d0461d6, State: Initialized, Role: FOLLOWER
I20260619 12:04:20.850399 23128 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.850739 23128 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:20.850759 23102 raft_consensus.cc:482] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 12:04:20.850803 23102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:20.851715 23102 raft_consensus.cc:504] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.852010 23102 leader_election.cc:288] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [CANDIDATE]: Term 1 election: Requested vote from peers 2c7377bd926446afa987f08b36956045 (127.0.203.253:42131), 4756a01cd940444581e7faf53e95cadf (127.0.203.252:41661)
I20260619 12:04:20.850703 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.852355 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:20.852473 23177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f02c34e9be964900a8cdd86e7d0461d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c7377bd926446afa987f08b36956045"
I20260619 12:04:20.852649 23177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:20.853857 23177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f02c34e9be964900a8cdd86e7d0461d6 in term 1.
I20260619 12:04:20.854981 23079 leader_election.cc:302] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c7377bd926446afa987f08b36956045, f02c34e9be964900a8cdd86e7d0461d6; no voters:
I20260619 12:04:20.855312 23312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:20.855464 23312 raft_consensus.cc:686] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [term 1 LEADER]: Becoming Leader. State: Replica: f02c34e9be964900a8cdd86e7d0461d6, State: Running, Role: LEADER
I20260619 12:04:20.855448 23252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f02c34e9be964900a8cdd86e7d0461d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4756a01cd940444581e7faf53e95cadf"
I20260619 12:04:20.855548 23312 consensus_queue.cc:237] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } }
I20260619 12:04:20.855583 23252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:20.855980 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f02c34e9be964900a8cdd86e7d0461d6. Latest consensus state: current_term: 1 leader_uuid: "f02c34e9be964900a8cdd86e7d0461d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.856050 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:20.856724 23252 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f02c34e9be964900a8cdd86e7d0461d6 in term 1.
I20260619 12:04:20.857759 23329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:20.858342 23329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:20.860839 23252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [term 1 FOLLOWER]: Refusing update from remote peer f02c34e9be964900a8cdd86e7d0461d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:04:20.861255 23327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:04:20.862605 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f02c34e9be964900a8cdd86e7d0461d6. Latest consensus state: current_term: 1 leader_uuid: "f02c34e9be964900a8cdd86e7d0461d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.862694 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:20.863256 23177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [term 1 FOLLOWER]: Refusing update from remote peer f02c34e9be964900a8cdd86e7d0461d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:04:20.863590 23327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:04:20.865011 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f02c34e9be964900a8cdd86e7d0461d6. Latest consensus state: current_term: 1 leader_uuid: "f02c34e9be964900a8cdd86e7d0461d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.865104 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:20.866212 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f02c34e9be964900a8cdd86e7d0461d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.866302 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:20.866300 23298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f02c34e9be964900a8cdd86e7d0461d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.866366 23298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:20.866443 23335 mvcc.cc:205] Tried to move back new op lower bound from 7298542226883612672 to 7298542226865061888. Current Snapshot: MvccSnapshot[applied={T|T < 7298542226883612672}]
I20260619 12:04:20.867094 23338 mvcc.cc:205] Tried to move back new op lower bound from 7298542226883612672 to 7298542226865061888. Current Snapshot: MvccSnapshot[applied={T|T < 7298542226883612672}]
I20260619 12:04:20.867936 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f02c34e9be964900a8cdd86e7d0461d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.868016 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:20.868399 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f02c34e9be964900a8cdd86e7d0461d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02c34e9be964900a8cdd86e7d0461d6" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 40019 } } peers { permanent_uuid: "2c7377bd926446afa987f08b36956045" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 42131 } } peers { permanent_uuid: "4756a01cd940444581e7faf53e95cadf" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41661 } } }
I20260619 12:04:20.868500 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:20.869169 23336 mvcc.cc:205] Tried to move back new op lower bound from 7298542226883612672 to 7298542226865061888. Current Snapshot: MvccSnapshot[applied={T|T < 7298542226883612672}]
I20260619 12:04:20.873327 23329 catalog_manager.cc:1347] Generated new cluster ID: 99b95de076634960b2254ba92edc706e
I20260619 12:04:20.873397 23329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:20.873826 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.854979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.0.203.253:42131)
0619 12:04:20.855056 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.855059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.864887 (+ 9828us) server_negotiation.cc:407] Connection header received
0619 12:04:20.865013 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.865014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.865027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.865058 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.866026 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.866823 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.867911 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.868227 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.868269 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.870344 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.870346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.870346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.870356 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.872233 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.872799 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.872801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.872801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.872867 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.873087 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.873088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.873088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.873240 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.873327 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.873649 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.873732 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:04:20.881840 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.857546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.0.203.254:40019)
0619 12:04:20.857601 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.857603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.874613 (+ 17010us) server_negotiation.cc:407] Connection header received
0619 12:04:20.874698 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.874699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.874710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.874736 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.875365 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.876094 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.877088 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.877370 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.877406 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.878816 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.878818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.878818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.878825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.880425 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.880898 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.880899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.880900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.880968 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.881189 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.881190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.881191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.881326 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.881459 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.881662 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.881750 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 12:04:20.884047 23329 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:20.884927 23329 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:20.890878 23329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f02c34e9be964900a8cdd86e7d0461d6: Generated new TSK 0
I20260619 12:04:20.891116 23329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:04:20.892215 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:20.856333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.0.203.252:41661)
0619 12:04:20.856378 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.856380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.870424 (+ 14044us) server_negotiation.cc:407] Connection header received
0619 12:04:20.870528 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.870530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.870543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.870575 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.871188 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.871933 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.875015 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:20.875349 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:20.875393 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:20.888542 (+ 13149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:20.888545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:20.888545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:20.888556 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:20.890493 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.890941 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.890942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.890943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.891003 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:20.891329 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:20.891330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:20.891331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:20.891487 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:20.891577 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:20.892042 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 12:04:20.892119 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 12:04:20.919257 23282 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.917563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.203.252:41661)
0619 12:04:20.917612 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.917614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.918577 (+ 963us) server_negotiation.cc:407] Connection header received
0619 12:04:20.918684 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.918685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.918697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.918729 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.919178 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52402: BlockingRecv error: recv got EOF from 127.0.0.1:52402 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 12:04:20.920256 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.917984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.0.203.253:42131)
0619 12:04:20.918027 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.918029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.919678 (+ 1649us) server_negotiation.cc:407] Connection header received
0619 12:04:20.919775 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.919776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.919788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.919816 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.920174 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55180: BlockingRecv error: recv got EOF from 127.0.0.1:55180 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 12:04:20.921173 23131 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.918321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.0.203.254:40019)
0619 12:04:20.918372 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.918374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.920631 (+ 2257us) server_negotiation.cc:407] Connection header received
0619 12:04:20.920779 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.920780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.920791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.920819 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.921113 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46702: BlockingRecv error: recv got EOF from 127.0.0.1:46702 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 12:04:20.923650 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.922909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.0.203.253:42131)
0619 12:04:20.922946 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.922948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.923140 (+ 192us) server_negotiation.cc:407] Connection header received
0619 12:04:20.923246 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.923247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.923268 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.923296 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.923604 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55196: BlockingRecv error: recv got EOF from 127.0.0.1:55196 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 12:04:20.926709 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.925982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.0.203.253:42131)
0619 12:04:20.926014 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.926016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.926240 (+ 224us) server_negotiation.cc:407] Connection header received
0619 12:04:20.926330 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.926331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.926353 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.926381 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.926665 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55202: BlockingRecv error: recv got EOF from 127.0.0.1:55202 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 12:04:20.930733 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.929910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.0.203.253:42131)
0619 12:04:20.929944 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.929946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.930202 (+ 256us) server_negotiation.cc:407] Connection header received
0619 12:04:20.930330 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.930331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.930350 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.930381 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.930690 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55212: BlockingRecv error: recv got EOF from 127.0.0.1:55212 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 12:04:20.936887 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.936077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.0.203.253:42131)
0619 12:04:20.936113 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.936115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.936322 (+ 207us) server_negotiation.cc:407] Connection header received
0619 12:04:20.936432 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.936434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.936444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.936473 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.936833 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55214: BlockingRecv error: recv got EOF from 127.0.0.1:55214 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 12:04:20.947232 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.946330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.0.203.253:42131)
0619 12:04:20.946390 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.946392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.946660 (+ 268us) server_negotiation.cc:407] Connection header received
0619 12:04:20.946766 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.946767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.946779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.946808 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.947168 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55222: BlockingRecv error: recv got EOF from 127.0.0.1:55222 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 12:04:20.965790 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.964794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.0.203.253:42131)
0619 12:04:20.964839 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.964841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.965165 (+ 324us) server_negotiation.cc:407] Connection header received
0619 12:04:20.965289 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.965291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.965303 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.965333 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:20.965725 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55234: BlockingRecv error: recv got EOF from 127.0.0.1:55234 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 12:04:21.000468 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:20.999366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.0.203.253:42131)
0619 12:04:20.999432 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:20.999436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:20.999763 (+ 327us) server_negotiation.cc:407] Connection header received
0619 12:04:20.999948 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:20.999949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:20.999962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:20.999993 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:21.000399 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55238: BlockingRecv error: recv got EOF from 127.0.0.1:55238 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 12:04:21.066947 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:21.066090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.0.203.253:42131)
0619 12:04:21.066177 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.066179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.066284 (+ 105us) server_negotiation.cc:407] Connection header received
0619 12:04:21.066387 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.066388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.066397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.066440 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:21.066897 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55250: BlockingRecv error: recv got EOF from 127.0.0.1:55250 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 12:04:21.197633 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:21.196612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.0.203.253:42131)
0619 12:04:21.196741 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.196744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.196894 (+ 150us) server_negotiation.cc:407] Connection header received
0619 12:04:21.197022 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.197023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.197037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.197110 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:21.197532 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 12:04:21.456036 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:21.455205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.0.203.253:42131)
0619 12:04:21.455288 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.455290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.455372 (+ 82us) server_negotiation.cc:407] Connection header received
0619 12:04:21.455455 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.455456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.455467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.455538 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:21.455968 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55270: BlockingRecv error: recv got EOF from 127.0.0.1:55270 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:04:21.789597 23295 catalog_manager.cc:1259] Loaded cluster ID: 99b95de076634960b2254ba92edc706e
I20260619 12:04:21.789700 23295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: loading cluster ID for follower catalog manager: success
I20260619 12:04:21.790531 23295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: acquiring CA information for follower catalog manager: success
I20260619 12:04:21.790932 23295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4756a01cd940444581e7faf53e95cadf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:04:21.841787 23309 catalog_manager.cc:1259] Loaded cluster ID: 99b95de076634960b2254ba92edc706e
I20260619 12:04:21.841894 23309 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: loading cluster ID for follower catalog manager: success
I20260619 12:04:21.842877 23309 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: acquiring CA information for follower catalog manager: success
I20260619 12:04:21.843246 23309 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c7377bd926446afa987f08b36956045: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:04:21.973243 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:21.969758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.0.203.253:42131)
0619 12:04:21.969887 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.969889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.969977 (+ 88us) server_negotiation.cc:407] Connection header received
0619 12:04:21.970037 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.970038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.970103 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.970189 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:21.970472 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.971033 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.971951 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.972162 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.972235 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:21.972504 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:21.972669 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:21.972743 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:21.973039 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 12:04:21.973068 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260619 12:04:21.978936 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:21.975695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.0.203.252:41661)
0619 12:04:21.975853 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.975856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.975921 (+ 65us) server_negotiation.cc:407] Connection header received
0619 12:04:21.976145 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.976146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.976196 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.976309 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:21.976458 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.977075 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.977862 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.978085 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.978165 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:21.978363 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:21.978512 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:21.978580 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:21.978817 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 12:04:21.978840 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260619 12:04:21.993196 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:21.989234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.0.203.253:42131)
0619 12:04:21.989292 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.989295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.989436 (+ 141us) server_negotiation.cc:407] Connection header received
0619 12:04:21.989553 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.989554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.989566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.989596 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:21.989828 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.990692 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.992227 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.992551 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.992626 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:21.992828 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:21.992865 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:21.992923 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:21.993056 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 12:04:21.993098 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:04:21.993372 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:21.989181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.203.252:41661)
0619 12:04:21.989252 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.989256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.989486 (+ 230us) server_negotiation.cc:407] Connection header received
0619 12:04:21.989633 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.989635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.989650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.989726 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:21.989956 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.990695 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.992084 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.992369 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.992421 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:21.993055 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:21.993094 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:21.993126 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:21.993241 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 12:04:21.993277 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 12:04:21.993508 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:21.989057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.0.203.254:40019)
0619 12:04:21.989264 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:21.989267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:21.989441 (+ 174us) server_negotiation.cc:407] Connection header received
0619 12:04:21.989562 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:21.989564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:21.989630 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:21.989755 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:21.989956 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.990741 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.992106 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:21.992446 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:21.992529 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:21.992813 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:21.992970 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:21.993035 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:21.993369 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 12:04:21.993400 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":117,"threads_started":1}
I20260619 12:04:21.995882 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 23060
I20260619 12:04:22.001742 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 23135
I20260619 12:04:22.007783 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 23210
2026-06-19T12:04:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2995 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:04:22 dist-test-slave-fbq8 krb5kdc[23424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:04:22 dist-test-slave-fbq8 krb5kdc[23424](info): set up 2 sockets
Jun 19 12:04:22 dist-test-slave-fbq8 krb5kdc[23424](info): commencing operation
krb5kdc: starting...
W20260619 12:04:23.073182 815 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/test-user.keytab.
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.203.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
Entry for principal kudu/127.0.203.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab.
I20260619 12:04:23.211882 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34973
--webserver_interface=127.0.203.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:34973,127.0.203.253:38075,127.0.203.252:41911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:04:23.298447 23437 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:04:23.298753 23437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:23.298785 23437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:23.298857 23437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:23.300211 23437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:23.300299 23437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:23.300323 23437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:23.300344 23437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:23.300375 23437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:23.302451 23437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:34973,127.0.203.253:38075,127.0.203.252:41911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.254.keytab
--principal=kudu/127.0.203.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.254:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.203.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:23.302868 23437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:23.303354 23437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:23.309424 23442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:23.309422 23443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:23.309439 23445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:23.309522 23437 server_base.cc:1034] running on GCE node
I20260619 12:04:23.310067 23437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:23.310457 23437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:23.311655 23437 hybrid_clock.cc:648] HybridClock initialized: now 1781870663311627 us; error 50 us; skew 500 ppm
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:23.318746 23437 init.cc:377] Logged in from keytab as kudu/127.0.203.254@KRBTEST.COM (short username kudu)
I20260619 12:04:23.319378 23437 webserver.cc:460] Webserver started at http://127.0.203.254:39309/ using document root <none> and password file <none>
I20260619 12:04:23.319653 23437 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:23.319706 23437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:23.319873 23437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:23.321213 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data/instance:
uuid: "80bc12641cc74921b7f2b1dd7b6542b7"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "a389e69c58a7c64be938a23ef0583bc9"
server_key_iv: "27ce6e0f2cdbf2f2c7f5f102cc7a3aea"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.321635 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/wal/instance:
uuid: "80bc12641cc74921b7f2b1dd7b6542b7"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "a389e69c58a7c64be938a23ef0583bc9"
server_key_iv: "27ce6e0f2cdbf2f2c7f5f102cc7a3aea"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.323335 23437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 12:04:23.324425 23452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:23.324872 23437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:23.324982 23437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/wal
uuid: "80bc12641cc74921b7f2b1dd7b6542b7"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "a389e69c58a7c64be938a23ef0583bc9"
server_key_iv: "27ce6e0f2cdbf2f2c7f5f102cc7a3aea"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.325052 23437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:23.360754 23437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:23.363754 23437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:23.363911 23437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:23.369469 23437 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.254:34973
I20260619 12:04:23.369516 23504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.254:34973 every 8 connection(s)
I20260619 12:04:23.370014 23437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/data/info.pb
I20260619 12:04:23.371268 23505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:23.373019 23505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:23.378691 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 23437
I20260619 12:04:23.378839 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-0/wal/instance
I20260619 12:04:23.379134 815 external_mini_cluster.cc:1428] Setting key 89a3ccb6728dec61c3128814da7211e3
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:23.386219 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.373945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.0.203.254:34973)
0619 12:04:23.374258 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.374262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.374290 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:04:23.374910 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.374914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.374938 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.375064 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.375751 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.376803 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.377818 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.378186 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.378288 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.381709 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.381717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.381717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.381726 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.383886 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.384674 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.384678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.384679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.384808 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.385081 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.385082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.385083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.385370 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.385484 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.385697 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.385878 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260619 12:04:23.386217 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.373873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:34973 (local address 127.0.0.1:37174)
0619 12:04:23.374103 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.374155 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.374691 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.375123 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.375126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.375422 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.375601 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.375605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.376943 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.376945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.377704 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.377707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.378324 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.378325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.378409 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.379155 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.379167 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.381567 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.384040 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.384058 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.384062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.384510 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.384894 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.384895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.384896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.385003 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.385453 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.385458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.385527 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.385660 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.385951 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":115,"thread_start_us":111,"threads_started":1}
I20260619 12:04:23.387458 23505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:23.388046 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.387874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51012)
0619 12:04:23.387938 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.387997 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:04:23.388259 23454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.203.253@KRBTEST.COM; defaulting to no policy
I20260619 12:04:23.388990 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.388823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51028)
0619 12:04:23.388920 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.388966 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 12:04:23.389187 23505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:38075: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab.
Entry for principal kudu/127.0.203.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab.
I20260619 12:04:23.412994 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:38075
--webserver_interface=127.0.203.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab
--principal=kudu/127.0.203.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:34973,127.0.203.253:38075,127.0.203.252:41911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:04:23.438446 23505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } attempt: 1
I20260619 12:04:23.439059 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.438911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51042)
0619 12:04:23.438987 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.439028 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:04:23.439647 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.439486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51046)
0619 12:04:23.439581 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.439623 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 12:04:23.439802 23505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:38075: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
W20260619 12:04:23.506230 23512 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:04:23.506500 23512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:23.506531 23512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:23.506579 23512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:23.507819 23512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:23.507894 23512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:23.507922 23512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:23.507956 23512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:23.507995 23512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:23.507999 23505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } attempt: 2
I20260619 12:04:23.508684 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.508468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51056)
0619 12:04:23.508555 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.508618 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:04:23.509120 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.509004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51070)
0619 12:04:23.509062 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.509097 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:04:23.509248 23505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.253:38075: Network error: Client connection negotiation failed: client connection to 127.0.203.253:38075: connect: Connection refused (error 111)
I20260619 12:04:23.510083 23512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:34973,127.0.203.253:38075,127.0.203.252:41911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.253.keytab
--principal=kudu/127.0.203.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.253:38075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.203.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:23.510378 23512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:23.510637 23512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:23.516062 23520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:23.516173 23518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:23.516083 23517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:23.516381 23512 server_base.cc:1034] running on GCE node
I20260619 12:04:23.516745 23512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:23.517166 23512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:23.518368 23512 hybrid_clock.cc:648] HybridClock initialized: now 1781870663518354 us; error 43 us; skew 500 ppm
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:23.521311 23512 init.cc:377] Logged in from keytab as kudu/127.0.203.253@KRBTEST.COM (short username kudu)
I20260619 12:04:23.521775 23512 webserver.cc:460] Webserver started at http://127.0.203.253:40617/ using document root <none> and password file <none>
I20260619 12:04:23.521988 23512 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:23.522032 23512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:23.522132 23512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:23.523336 23512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data/instance:
uuid: "8643f37c959a46f9a4dd781437049b77"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "3c9bfd1d75b79c1d1e77fc66e0f0ef09"
server_key_iv: "7ecd4bf9aa5629c8a8585277c0e6f888"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.523679 23512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/wal/instance:
uuid: "8643f37c959a46f9a4dd781437049b77"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "3c9bfd1d75b79c1d1e77fc66e0f0ef09"
server_key_iv: "7ecd4bf9aa5629c8a8585277c0e6f888"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.526638 23512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:04:23.527866 23527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:23.528318 23512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:23.528420 23512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/wal
uuid: "8643f37c959a46f9a4dd781437049b77"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "3c9bfd1d75b79c1d1e77fc66e0f0ef09"
server_key_iv: "7ecd4bf9aa5629c8a8585277c0e6f888"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.528970 23512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:23.541882 23512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:23.544054 23512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:23.544183 23512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:23.549475 23512 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.253:38075
I20260619 12:04:23.549624 23579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.253:38075 every 8 connection(s)
I20260619 12:04:23.549978 23512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/data/info.pb
I20260619 12:04:23.551132 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:23.552883 23580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:23.560119 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 23512
I20260619 12:04:23.560289 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-1/wal/instance
I20260619 12:04:23.560546 815 external_mini_cluster.cc:1428] Setting key 16b1d7375f9db637345dd64ccadac523
I20260619 12:04:23.564592 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.553718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.203.254:34973)
0619 12:04:23.553862 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.553871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.554093 (+ 222us) server_negotiation.cc:407] Connection header received
0619 12:04:23.554783 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.554784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.554797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.554854 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.555560 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.556109 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.557254 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.557478 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.557571 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.560543 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.560544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.560545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.560553 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.562516 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.563280 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.563281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.563282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.563339 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.563712 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.563713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.563714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.563917 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.564001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.564325 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.564432 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 12:04:23.564693 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.553616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:34973 (local address 127.0.0.1:37178)
0619 12:04:23.553905 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.553964 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.554657 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.554952 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.554956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.555243 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.555463 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.555467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.556208 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.556209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.557163 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.557165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.557600 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.557601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.557743 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.558372 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.558382 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.560394 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.562658 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.562673 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.562677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.563168 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.563428 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.563429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.563430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.563601 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.564005 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.564011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.564163 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.564273 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.564438 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":113,"threads_started":1}
I20260619 12:04:23.565541 23580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.203.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.203.252@KRBTEST.COM" created.
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:04:23.577554 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.565918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51072)
0619 12:04:23.565973 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.566189 (+ 216us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.566255 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.566404 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.566405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.566584 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.566640 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.566642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.567528 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.567529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.568032 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.568034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.568530 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.568530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.568596 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.569142 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.569153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.573734 (+ 4581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.575953 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.575955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.575956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.576246 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.576519 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.576520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.576520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.576602 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.576985 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.576986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.577059 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.577149 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.577297 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 12:04:23.577560 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.566018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.0.203.253:38075)
0619 12:04:23.566196 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.566202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.566253 (+ 51us) server_negotiation.cc:407] Connection header received
0619 12:04:23.566315 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.566317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.566331 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.566402 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.566722 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.567437 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.568117 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.568451 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.568535 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.573872 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.573884 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.573884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.573895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.575847 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.576358 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.576361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.576361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.576424 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.576725 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.576726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.576726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.576901 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.576975 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.577232 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.577384 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260619 12:04:23.578418 23580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:23.578941 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.578728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52360)
0619 12:04:23.578798 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.578895 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:04:23.579151 23531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
I20260619 12:04:23.579686 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.579550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52364)
0619 12:04:23.579619 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.579662 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 12:04:23.579833 23580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41911: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.203.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab.
Entry for principal kudu/127.0.203.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab.
I20260619 12:04:23.592193 815 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwGpnVV/build/release/bin/kudu
/tmp/dist-test-taskwGpnVV/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:41911
--webserver_interface=127.0.203.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab
--principal=kudu/127.0.203.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.203.212:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.203.254:34973,127.0.203.253:38075,127.0.203.252:41911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:04:23.600414 23505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } attempt: 3
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:04:23.608991 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.601054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.0.203.253:38075)
0619 12:04:23.601147 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.601150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.601161 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:04:23.601199 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.601200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.601210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.601277 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.601685 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.602252 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.602988 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.603236 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.603319 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.605752 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.605754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.605754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.605761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.607346 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.607993 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.607995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.607995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.608083 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.608367 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.608369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.608369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.608515 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.608589 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.608840 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.608909 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:04:23.609112 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.600929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51088)
0619 12:04:23.601004 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.601019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.601076 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.601282 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.601283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.601507 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.601588 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.601590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.602345 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.602347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.602902 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.602903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.603313 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.603314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.603399 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.603934 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.603939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.605609 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.607452 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.607454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.607455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.607822 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.608176 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.608177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.608177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.608260 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.608609 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.608611 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.608713 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.608812 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.608959 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 12:04:23.609603 23505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:23.610041 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.609903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52378)
0619 12:04:23.609968 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.610013 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 12:04:23.610445 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.610340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52390)
0619 12:04:23.610399 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.610427 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:04:23.610623 23505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41911: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
I20260619 12:04:23.629091 23580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } attempt: 1
I20260619 12:04:23.629727 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.629507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52406)
0619 12:04:23.629602 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.629671 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 12:04:23.630213 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.630091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52408)
0619 12:04:23.630149 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.630186 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:04:23.630379 23580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41911: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
I20260619 12:04:23.641826 23505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } attempt: 1
I20260619 12:04:23.642446 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.642205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52420)
0619 12:04:23.642294 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.642363 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:04:23.642915 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.642807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52430)
0619 12:04:23.642862 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.642895 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:04:23.643042 23505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41911: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
W20260619 12:04:23.686363 23587 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:04:23.686903 23587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:04:23.687062 23587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:04:23.687100 23587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:04:23.688426 23587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:04:23.688484 23587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:04:23.688500 23587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:04:23.688513 23587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:04:23.688529 23587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:04:23.690299 23587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.203.212:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.203.254:34973,127.0.203.253:38075,127.0.203.252:41911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/krb5kdc/kudu_127.0.203.252.keytab
--principal=kudu/127.0.203.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.203.252:41911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.203.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 10:56:02 UTC on ccycloud.quasar-wyewmr.root.comops.site
build id 4455436
I20260619 12:04:23.690594 23587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:04:23.690891 23587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:04:23.696705 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:04:23.696671 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:23.696944 23587 server_base.cc:1034] running on GCE node
W20260619 12:04:23.696722 23592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:04:23.697439 23587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:04:23.697798 23587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:04:23.698576 23580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } attempt: 2
I20260619 12:04:23.698987 23587 hybrid_clock.cc:648] HybridClock initialized: now 1781870663698958 us; error 53 us; skew 500 ppm
I20260619 12:04:23.699179 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.698944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52440)
0619 12:04:23.699047 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.699132 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 12:04:23.699673 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.699579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52442)
0619 12:04:23.699625 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.699651 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 12:04:23.699918 23580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41911: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:04:23.702118 23587 init.cc:377] Logged in from keytab as kudu/127.0.203.252@KRBTEST.COM (short username kudu)
I20260619 12:04:23.702677 23587 webserver.cc:460] Webserver started at http://127.0.203.252:33627/ using document root <none> and password file <none>
I20260619 12:04:23.702939 23587 fs_manager.cc:359] Metadata directory not provided
I20260619 12:04:23.703009 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:04:23.703177 23587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:04:23.704403 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data/instance:
uuid: "8f790131b92742ef8a604874039240de"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "d9f1de9353fbe4fd14b03ae2bf35804b"
server_key_iv: "16d6fd99eff3be7328bf72240a6d2a5e"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.704826 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/wal/instance:
uuid: "8f790131b92742ef8a604874039240de"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "d9f1de9353fbe4fd14b03ae2bf35804b"
server_key_iv: "16d6fd99eff3be7328bf72240a6d2a5e"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.706318 23587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:23.707216 23602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:04:23.707491 23587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:04:23.707604 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data,/tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/wal
uuid: "8f790131b92742ef8a604874039240de"
format_stamp: "Formatted at 2026-06-19 12:04:23 on dist-test-slave-fbq8"
server_key: "d9f1de9353fbe4fd14b03ae2bf35804b"
server_key_iv: "16d6fd99eff3be7328bf72240a6d2a5e"
server_key_version: "encryptionkey@0"
I20260619 12:04:23.707683 23587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:04:23.718272 23505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } attempt: 2
I20260619 12:04:23.719069 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.718744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52454)
0619 12:04:23.718861 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.719021 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 12:04:23.719623 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.719510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52466)
0619 12:04:23.719565 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.719601 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 12:04:23.719758 23505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.203.252:41911: Network error: Client connection negotiation failed: client connection to 127.0.203.252:41911: connect: Connection refused (error 111)
I20260619 12:04:23.740109 23587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:04:23.743255 23587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:04:23.743396 23587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:04:23.749222 23587 rpc_server.cc:307] RPC server started. Bound to: 127.0.203.252:41911
I20260619 12:04:23.749341 23654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.203.252:41911 every 8 connection(s)
I20260619 12:04:23.749754 23587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/data/info.pb
I20260619 12:04:23.750919 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:04:23.752790 23655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:04:23.759366 815 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwGpnVV/build/release/bin/kudu as pid 23587
I20260619 12:04:23.759536 815 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwGpnVV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781870380032316-815-0/minicluster-data/master-2/wal/instance
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
I20260619 12:04:23.759845 815 external_mini_cluster.cc:1428] Setting key f3dbf4b979d1ced73e9a10c8951faa61
I20260619 12:04:23.763918 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.753705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.0.203.254:34973)
0619 12:04:23.753815 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.753818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.754116 (+ 298us) server_negotiation.cc:407] Connection header received
0619 12:04:23.754649 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.754650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.754661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.754727 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.755501 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.756008 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.757205 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.757552 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.757626 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.760347 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.760349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.760349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.760358 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.762245 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.762793 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.762795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.762796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.762879 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.763159 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.763160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.763160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.763300 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.763381 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.763717 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.763816 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 12:04:23.764140 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.753677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.254:34973 (local address 127.0.0.1:37184)
0619 12:04:23.753936 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.753978 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.754521 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.754855 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.754858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.755142 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.755354 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.755358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.756120 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.756121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.757062 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.757064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.757672 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.757673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.757807 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.758420 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.758429 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.760204 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.762346 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.762352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.762354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.762677 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.762952 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.762952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.762953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.763075 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.763370 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.763376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.763535 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.763676 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.763866 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":127,"threads_started":1}
I20260619 12:04:23.764910 23655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:04:23.769582 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.760319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.0.203.252:41911)
0619 12:04:23.760548 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.760553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.760583 (+ 30us) server_negotiation.cc:407] Connection header received
0619 12:04:23.760616 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.760619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.760666 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.760753 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.761565 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.762599 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.763415 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.763744 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.763829 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.766524 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.766532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.766533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.766542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.768090 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.768454 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.768457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.768458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.768526 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.768740 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.768742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.768742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.768943 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.769124 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.769224 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.769315 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":132,"threads_started":1}
I20260619 12:04:23.772776 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.765367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.203.253:38075)
0619 12:04:23.765453 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.765455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.765464 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:04:23.765494 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.765495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.765506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.765582 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.765886 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.766468 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.767525 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.767742 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.767828 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.770085 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.770087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.770087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.770097 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.771574 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.771943 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.771944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.771944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.772001 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.772204 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.772205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.772205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.772328 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.772404 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.772580 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.772682 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:04:23.772776 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.765248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.253:38075 (local address 127.0.0.1:51090)
0619 12:04:23.765320 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.765334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.765407 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.765589 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.765590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.765746 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.765801 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.765803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.766548 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.766550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.767251 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.767253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.767823 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.767824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.767906 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.768305 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.768310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.769776 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.771682 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.771683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.771684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.771855 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.772069 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.772070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.772070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.772127 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.772397 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.772398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.772443 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.772551 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.772682 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 12:04:23.773299 23655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.254@KRBTEST.COM
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.252@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:04:23.778458 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.770490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.0.203.254:34973)
0619 12:04:23.770602 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.770604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.770612 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:04:23.770637 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.770638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.770654 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.770760 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.771266 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.771740 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.772406 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.772617 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.772748 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.775411 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.775413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.775414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.775421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.777182 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.777568 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.777569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.777570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.777630 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.777832 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.777832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.777833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.777980 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.778146 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.778316 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.778382 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 12:04:23.780700 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.773602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52476)
0619 12:04:23.773669 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.773686 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.773726 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.773881 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.773882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.774007 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.774047 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.774049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.774811 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.774812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.775416 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.775419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.775911 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.775912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.775979 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.776356 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.776361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.777664 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.779532 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.779533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.779534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.779820 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.780039 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.780040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.780041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.780115 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.780400 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.780401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.780439 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.780495 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.780574 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 12:04:23.780836 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.773649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.0.203.252:41911)
0619 12:04:23.773695 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.773697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.773720 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:04:23.773793 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.773794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.773808 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.773874 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.774121 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.774725 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.775485 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.775790 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.775884 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.777756 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.777758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.777759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.777766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.779445 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.779911 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.779912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.779913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.779967 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.780184 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.780185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.780185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.780325 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.780398 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.780550 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.780671 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 12:04:23.782703 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: Bootstrap starting.
I20260619 12:04:23.783836 23655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:23.784178 23655 log.cc:824] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.203.253@KRBTEST.COM
I20260619 12:04:23.785056 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: No bootstrap required, opened a new log
I20260619 12:04:23.786527 23655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.786686 23655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f790131b92742ef8a604874039240de, State: Initialized, Role: FOLLOWER
I20260619 12:04:23.786782 23655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.787088 23661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.787153 23655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:23.787196 23661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:23.788095 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.779495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.0.203.253:38075)
0619 12:04:23.779542 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.779544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.779553 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:04:23.779577 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.779578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.779588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.779647 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.780300 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.781086 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.782023 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.782246 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.782305 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.785054 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.785056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.785057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.785065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.786602 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.787000 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.787001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.787001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.787059 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.787266 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.787267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.787268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.787405 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.787518 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.787802 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.787896 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 12:04:23.789191 23672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:23.789258 23672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:23.789884 815 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:04:23.791103 23580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } attempt: 3
I20260619 12:04:23.790943 23479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "80bc12641cc74921b7f2b1dd7b6542b7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37186
I20260619 12:04:23.789757 23587 master_runner.cc:186] Error getting master registration for 127.0.203.254:34973: OK, instance_id { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" instance_seqno: 1781870663367746 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:04:23.793243 23587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.253@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:04:23.800283 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.791388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52492)
0619 12:04:23.791483 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.791495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.791533 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.792315 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.792316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.792517 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.792570 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.792572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.793381 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.793382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.794176 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.794177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.794548 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.794549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.794602 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.795226 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.795231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.796915 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.798914 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.798916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.798917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.799186 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.799431 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.799432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.799433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.799507 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.799887 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.799889 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.799950 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.800036 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.800170 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 12:04:23.800285 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.791608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.0.203.252:41911)
0619 12:04:23.792021 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.792023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.792031 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:04:23.792098 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.792099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.792111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.792190 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.792670 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.793271 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.794274 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.794472 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.794551 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.797052 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.797054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.797054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.797062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.798803 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.799276 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.799277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.799278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.799343 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.799597 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.799598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.799599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.799805 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.799876 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.800078 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.800173 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260619 12:04:23.802242 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: Bootstrap starting.
I20260619 12:04:23.802897 23580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:23.803190 23580 log.cc:824] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:23.803951 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: No bootstrap required, opened a new log
I20260619 12:04:23.805492 23580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.805630 23580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8643f37c959a46f9a4dd781437049b77, State: Initialized, Role: FOLLOWER
I20260619 12:04:23.805727 23580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.806066 23675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.806142 23580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:23.806162 23675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 12:04:23.807961 23686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:04:23.808023 23686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:04:23.808243 23512 master_runner.cc:186] Error getting master registration for 127.0.203.254:34973: OK, instance_id { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" instance_seqno: 1781870663367746 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:04:23.811311 23512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:04:23.818934 23505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } attempt: 3
Jun 19 12:04:23 dist-test-slave-fbq8 krb5kdc[23424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781870663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.203.254@KRBTEST.COM for kudu/127.0.203.252@KRBTEST.COM
I20260619 12:04:23.826885 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.819367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.0.203.252:41911)
0619 12:04:23.819418 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.819421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.819430 (+ 9us) server_negotiation.cc:407] Connection header received
0619 12:04:23.819460 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.819461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.819472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.819525 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.819869 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.820351 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.821358 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.821551 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.821645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.823913 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.823915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.823915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.823923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.825633 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.826011 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.826012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.826013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.826102 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.826304 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.826305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.826306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.826449 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.826518 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.826707 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.826778 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:04:23.826876 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.819270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.203.252:41911 (local address 127.0.0.1:52496)
0619 12:04:23.819328 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 12:04:23.819341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:04:23.819385 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:04:23.819531 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:04:23.819532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:04:23.819738 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:04:23.819796 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.819798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.820451 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.820452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.821288 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:04:23.821290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.821644 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.821645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:04:23.821701 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.822212 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:04:23.822217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:04:23.823815 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:04:23.825722 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.825723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.825723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.825907 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.826172 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:04:23.826172 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:04:23.826173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:04:23.826233 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:04:23.826519 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:04:23.826520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:04:23.826569 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 12:04:23.826658 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:04:23.826768 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 12:04:23.829252 23505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7: Bootstrap starting.
I20260619 12:04:23.830057 23505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7: Neither blocks nor log segments found. Creating new log.
I20260619 12:04:23.830396 23505 log.cc:824] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:04:23.831260 23505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7: No bootstrap required, opened a new log
I20260619 12:04:23.832723 23505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.832875 23505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bc12641cc74921b7f2b1dd7b6542b7, State: Initialized, Role: FOLLOWER
I20260619 12:04:23.832962 23505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.833348 23479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 12:04:23.833470 23479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:23.834117 23479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.834400 23479 leader_election.cc:288] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8643f37c959a46f9a4dd781437049b77 (127.0.203.253:38075), 8f790131b92742ef8a604874039240de (127.0.203.252:41911)
I20260619 12:04:23.833321 23505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:04:23.834862 23554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8643f37c959a46f9a4dd781437049b77"
I20260619 12:04:23.835057 23554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:23.836122 23554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bc12641cc74921b7f2b1dd7b6542b7 in term 1.
I20260619 12:04:23.833312 23689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.836905 23689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:23.837147 23454 leader_election.cc:302] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80bc12641cc74921b7f2b1dd7b6542b7, 8643f37c959a46f9a4dd781437049b77; no voters:
I20260619 12:04:23.837278 23689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:04:23.837430 23689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [term 1 LEADER]: Becoming Leader. State: Replica: 80bc12641cc74921b7f2b1dd7b6542b7, State: Running, Role: LEADER
I20260619 12:04:23.837517 23689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } }
I20260619 12:04:23.837968 23689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bc12641cc74921b7f2b1dd7b6542b7. Latest consensus state: current_term: 1 leader_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.838032 23689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:23.837944 23629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8f790131b92742ef8a604874039240de"
I20260619 12:04:23.838078 23629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:04:23.838222 23707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:04:23.838739 23707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:04:23.839138 23629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80bc12641cc74921b7f2b1dd7b6542b7 in term 1.
I20260619 12:04:23.841667 23554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [term 1 FOLLOWER]: Refusing update from remote peer 80bc12641cc74921b7f2b1dd7b6542b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:04:23.842216 23437 master_runner.cc:263] No leader master found from master 80bc12641cc74921b7f2b1dd7b6542b7
I20260619 12:04:23.842262 23437 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:04:23.842509 23629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [term 1 FOLLOWER]: Refusing update from remote peer 80bc12641cc74921b7f2b1dd7b6542b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:04:23.842809 23689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:04:23.842964 23689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:04:23.844295 23675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bc12641cc74921b7f2b1dd7b6542b7. Latest consensus state: current_term: 1 leader_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.844487 23675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:23.845968 23661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bc12641cc74921b7f2b1dd7b6542b7. Latest consensus state: current_term: 1 leader_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.846091 23661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:23.848248 23689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.848354 23689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:23.848891 23675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.848985 23675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:23.849270 23709 mvcc.cc:205] Tried to move back new op lower bound from 7298542239092842496 to 7298542239079452672. Current Snapshot: MvccSnapshot[applied={T|T < 7298542239092842496}]
I20260619 12:04:23.849773 23711 mvcc.cc:205] Tried to move back new op lower bound from 7298542239092842496 to 7298542239079452672. Current Snapshot: MvccSnapshot[applied={T|T < 7298542239092842496}]
I20260619 12:04:23.850816 23707 catalog_manager.cc:1347] Generated new cluster ID: a9cd2055e9454a9dae7adffc08f58356
I20260619 12:04:23.850878 23707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:04:23.852404 23705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.852500 23705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:04:23.853224 23661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80bc12641cc74921b7f2b1dd7b6542b7" member_type: VOTER last_known_addr { host: "127.0.203.254" port: 34973 } } peers { permanent_uuid: "8643f37c959a46f9a4dd781437049b77" member_type: VOTER last_known_addr { host: "127.0.203.253" port: 38075 } } peers { permanent_uuid: "8f790131b92742ef8a604874039240de" member_type: VOTER last_known_addr { host: "127.0.203.252" port: 41911 } } }
I20260619 12:04:23.853318 23661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:04:23.853606 23713 mvcc.cc:205] Tried to move back new op lower bound from 7298542239092842496 to 7298542239079452672. Current Snapshot: MvccSnapshot[applied={T|T < 7298542239092842496}]
I20260619 12:04:23.858872 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.840681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.203.252:41911)
0619 12:04:23.840738 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.840740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.846925 (+ 6185us) server_negotiation.cc:407] Connection header received
0619 12:04:23.847040 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.847041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.847052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.847083 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.847905 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.848944 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.850371 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.850717 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.850763 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.855752 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.855755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.855755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.855765 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.857588 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.858059 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.858061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.858061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.858111 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.858290 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.858290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.858291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.858413 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.858486 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.858687 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.858762 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:04:23.866990 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.846626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.0.203.253:38075)
0619 12:04:23.846686 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.846689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.852799 (+ 6110us) server_negotiation.cc:407] Connection header received
0619 12:04:23.853911 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.853913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.853926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.853958 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.855902 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.856714 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.860836 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.861192 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.861238 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.863923 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.863925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.863925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.863934 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.865589 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.866173 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.866174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.866175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.866226 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.866420 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.866421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.866421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.866545 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.866625 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.866807 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.866891 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:04:23.873677 23707 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:04:23.874229 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:23.840952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.0.203.254:34973)
0619 12:04:23.841022 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.841024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.852242 (+ 11218us) server_negotiation.cc:407] Connection header received
0619 12:04:23.852943 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.852944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.852960 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.852994 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.858042 (+ 5048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.858798 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.860778 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:23.861091 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:23.861173 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:23.862994 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:04:23.862996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:04:23.862996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:04:23.863005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:04:23.864845 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.865969 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.865971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.865971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.866042 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:04:23.873516 (+ 7474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:04:23.873517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:04:23.873518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:04:23.873677 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:04:23.873798 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:23.874013 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 12:04:23.874119 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:04:23.874504 23707 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:04:23.890899 23707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80bc12641cc74921b7f2b1dd7b6542b7: Generated new TSK 0
I20260619 12:04:23.891194 23707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:04:23.918773 23659 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.917533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.0.203.252:41911)
0619 12:04:23.917590 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.917593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.917974 (+ 381us) server_negotiation.cc:407] Connection header received
0619 12:04:23.918117 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.918119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.918133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.918168 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.918689 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52506: BlockingRecv error: recv got EOF from 127.0.0.1:52506 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 12:04:23.921870 23508 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.920924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.0.203.254:34973)
0619 12:04:23.920971 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.920973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.921311 (+ 338us) server_negotiation.cc:407] Connection header received
0619 12:04:23.921410 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.921411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.921422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.921452 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.921796 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37208: BlockingRecv error: recv got EOF from 127.0.0.1:37208 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 12:04:23.922631 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.921188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.0.203.253:38075)
0619 12:04:23.921231 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.921233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.922143 (+ 910us) server_negotiation.cc:407] Connection header received
0619 12:04:23.922233 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.922234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.922245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.922275 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.922569 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51120: BlockingRecv error: recv got EOF from 127.0.0.1:51120 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 12:04:23.925429 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.924245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.0.203.253:38075)
0619 12:04:23.924283 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.924285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.924471 (+ 186us) server_negotiation.cc:407] Connection header received
0619 12:04:23.924570 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.924571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.924589 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.924618 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.925333 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51134: BlockingRecv error: recv got EOF from 127.0.0.1:51134 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 12:04:23.929030 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.928061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.0.203.253:38075)
0619 12:04:23.928116 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.928118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.928393 (+ 275us) server_negotiation.cc:407] Connection header received
0619 12:04:23.928495 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.928496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.928508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.928538 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.928968 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51146: BlockingRecv error: recv got EOF from 127.0.0.1:51146 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 12:04:23.933615 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.932515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.0.203.253:38075)
0619 12:04:23.932572 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.932575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.932965 (+ 390us) server_negotiation.cc:407] Connection header received
0619 12:04:23.933080 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.933081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.933093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.933127 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.933517 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51150: BlockingRecv error: recv got EOF from 127.0.0.1:51150 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 12:04:23.940007 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.939065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.0.203.253:38075)
0619 12:04:23.939106 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.939108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.939381 (+ 273us) server_negotiation.cc:407] Connection header received
0619 12:04:23.939491 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.939492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.939505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.939556 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.939949 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51156: BlockingRecv error: recv got EOF from 127.0.0.1:51156 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 12:04:23.950485 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.949522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.0.203.253:38075)
0619 12:04:23.949578 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.949580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.949869 (+ 289us) server_negotiation.cc:407] Connection header received
0619 12:04:23.949981 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.949982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.949996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.950029 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.950423 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51160: BlockingRecv error: recv got EOF from 127.0.0.1:51160 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 12:04:23.969285 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:23.968144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.203.253:38075)
0619 12:04:23.968195 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:23.968198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:23.968512 (+ 314us) server_negotiation.cc:407] Connection header received
0619 12:04:23.968672 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:23.968674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:23.968688 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:23.968740 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:23.969206 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51176: BlockingRecv error: recv got EOF from 127.0.0.1:51176 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 12:04:24.004268 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:24.003226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.0.203.253:38075)
0619 12:04:24.003283 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.003286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.003589 (+ 303us) server_negotiation.cc:407] Connection header received
0619 12:04:24.003712 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.003713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.003726 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.003758 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:24.004188 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51178: BlockingRecv error: recv got EOF from 127.0.0.1:51178 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 12:04:24.071008 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:24.070041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.0.203.253:38075)
0619 12:04:24.070175 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.070177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.070289 (+ 112us) server_negotiation.cc:407] Connection header received
0619 12:04:24.070354 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.070355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.070369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.070436 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:24.070878 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51190: BlockingRecv error: recv got EOF from 127.0.0.1:51190 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 12:04:24.201519 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:24.200568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.0.203.253:38075)
0619 12:04:24.200690 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.200694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.200812 (+ 118us) server_negotiation.cc:407] Connection header received
0619 12:04:24.200857 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.200859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.200874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.200981 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:24.201431 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51200: BlockingRecv error: recv got EOF from 127.0.0.1:51200 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 12:04:24.460078 23584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:04:24.459190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.0.203.253:38075)
0619 12:04:24.459310 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.459313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.459328 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:04:24.459432 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.459433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.459446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.459513 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:04:24.460002 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60752: BlockingRecv error: recv got EOF from 127.0.0.1:60752 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 12:04:24.789748 23672 catalog_manager.cc:1259] Loaded cluster ID: a9cd2055e9454a9dae7adffc08f58356
I20260619 12:04:24.789872 23672 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: loading cluster ID for follower catalog manager: success
I20260619 12:04:24.790767 23672 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: acquiring CA information for follower catalog manager: success
I20260619 12:04:24.791085 23672 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f790131b92742ef8a604874039240de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:04:24.808679 23686 catalog_manager.cc:1259] Loaded cluster ID: a9cd2055e9454a9dae7adffc08f58356
I20260619 12:04:24.808851 23686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: loading cluster ID for follower catalog manager: success
I20260619 12:04:24.809790 23686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: acquiring CA information for follower catalog manager: success
I20260619 12:04:24.810163 23686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8643f37c959a46f9a4dd781437049b77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:04:24.976770 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:24.973791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.0.203.253:38075)
0619 12:04:24.974012 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.974014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.974027 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:04:24.974140 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.974141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.974208 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.974300 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:24.974483 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.975007 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.975819 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.976081 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.976145 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:24.976341 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:24.976465 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:24.976566 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:24.976596 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 12:04:24.976619 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260619 12:04:24.982309 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:24.978907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.0.203.252:41911)
0619 12:04:24.979035 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.979043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.979078 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:04:24.979125 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.979126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.979174 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.979255 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:24.979368 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.979935 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.980930 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.981182 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.981252 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:24.981599 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:24.981727 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:24.981813 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:24.982085 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 12:04:24.982169 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260619 12:04:24.988126 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:24.984801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.203.252:41911)
0619 12:04:24.984852 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.984855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.984923 (+ 68us) server_negotiation.cc:407] Connection header received
0619 12:04:24.985044 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.985045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.985058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.985125 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:24.985348 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.985957 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.987103 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.987425 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.987468 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:24.987729 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:24.987768 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:24.987835 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:24.987996 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 12:04:24.988024 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:04:24.988456 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:24.984975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.0.203.253:38075)
0619 12:04:24.985043 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.985046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.985138 (+ 92us) server_negotiation.cc:407] Connection header received
0619 12:04:24.985179 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.985181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.985198 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.985280 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:24.985515 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.986253 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.987540 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.987831 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.987917 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:24.988140 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:24.988173 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:24.988201 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:24.988320 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 12:04:24.988349 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:04:24.988731 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:04:24.984630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.0.203.254:34973)
0619 12:04:24.984899 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 12:04:24.984901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 12:04:24.984909 (+ 8us) server_negotiation.cc:407] Connection header received
0619 12:04:24.985092 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:04:24.985094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:04:24.985154 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:04:24.985259 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:04:24.985417 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.986248 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.987328 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:04:24.987721 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:04:24.987799 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:04:24.988040 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:04:24.988160 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:04:24.988571 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0619 12:04:24.988592 (+ 21us) server_negotiation.cc:299] Negotiation successful
0619 12:04:24.988619 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":66,"threads_started":1}
I20260619 12:04:24.990870 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 23437
I20260619 12:04:24.997963 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 23512
I20260619 12:04:25.004189 815 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwGpnVV/build/release/bin/kudu with pid 23587
2026-06-19T12:04:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2995 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5990 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284977 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST